Depth measurement through display

ABSTRACT

Disclosed herein is a display device includingan illumination source for projecting an illumination pattern including a plurality of illumination features on a scene;an optical sensor for determining a first image including a plurality of reflection features;a translucent display, where the illumination source and the optical sensor are placed in a direction of propagation of the illumination pattern in front of the display; andan evaluation device configured for evaluating the first image by identifying and sorting the reflection features with respect to brightness, each reflection feature including a beam profile, determining a longitudinal coordinate for each reflection feature by analyzing their beam profiles,unambiguously matching reflection features with corresponding illumination features using the longitudinal coordinate classifying a reflection feature as a real feature or a false feature, rejecting the false features, and generating a depth map for the real features using the longitudinal coordinate.

FIELD OF THE INVENTION

The invention relates to a display device and a method for depthmeasurement through a translucent display and various uses of thedisplay device. The devices, methods and uses according to the presentinvention specifically may be employed for example in various areas ofdaily life, security technology, gaming, traffic technology, productiontechnology, photography such as digital photography or video photographyfor arts, documentation or technical purposes, safety technology,information technology, agriculture, crop protection, maintenance,cosmetics, medical technology or in the sciences. However, otherapplications are also possible.

PRIOR ART

Several display devices are known. Recent developments for devices witha display show that the display area should cover the whole space thatis available and the frame surrounding the display should be as small aspossible. This results in that electronic components and sensors, e.g.front facing camera, flashlight, proximity sensor and even 3D imagingsensors, cannot be arranged within the frame any longer but have to beplaced under the display. However, most common 3D imaging techniques andsystems such as 3D imaging system based on structured light or 3D-timeof flight (ToF) cannot be placed under the display without more ado.

Until now, it is not known that a 3D imaging system based on structuredlight or 3D-ToF works under a display, i.e. without making empty windowsthat do not contain any microcircuits and/or microwiring, for placingthe components or devices of the 3D imaging system to “see” throughthese windows.

For structured light, the main problem is the microstructure of themicrocircuits and/or microwiring of the transparent display and,consequently, the low light transmission through the display. Thismicrostructure results from the electrode matrix for addressing thesingle pixels. Also, the pixels itself represent an inverted gratingbecause the metal cathode of the single pixel is not transparent. Inprincipal the display structure could be made transparent or translucentas a whole, including the electrodes, by using specific materials untilnow there is no transparent or translucent display which does not have agrating like microstructure.

Structured-light based 3D imagers are based on projecting a point cloud,with several thousand points and with well know patterns, into ascenery. The microstructure of the transparent or translucent displayworks like a diffraction grating structure for laser light. As most ofthe projectors of structured light imagers are based on a laser sourcethat projects a well-defined dot pattern, this pattern experiences agrating effect of the display and every single spot of the dot patternwill show higher diffraction orders. This has a devastating effect for astructured light imager, because the additional and unwanted pointscaused by the grating structure make it highly complicated for itsalgorithm to retrieve the original expected patterns.

Furthermore, the number of projection points used for traditionalstructured light imagers are rather high. As a transparent display has avery low light transmission, e.g. even in the infrared (IR) at 850 nmand 940 nm which are the typical wavelength for 3D-imagers, very highoutput powers are needed for the structured light projectors to getenough power through the display which could be detected by the imager,which also must be located under the display which leads to anadditional light absorption. The combination of a high number of pointsand a low light transmission may lead to a low ambient light robustness.

For 3D-ToF sensors, the reflections on the display surfaces, which leadto multiple reflections, as well as the difference for delays when thelight passes through the display, different display structures havedifferent refractive indices, and prevents robust functionality whenused behind a display. Furthermore, 3D-ToF sensors also need a highamount of light to illuminate the scenery. In addition, illuminationshould be homogeneous. The low light transmission of the display makesit hard to provide enough light and the grating structure influences thehomogeneity of the illumination.

Common 3D sensing systems have problems to measure through transparentdisplays. Current devices use notches in the display. By that way, thesensors are not disturbed by the diffractive optical effects.

DE 20 2018 003 644 U1 describes a portable electronic device,comprising: a bottom wall and side walls defining a cavity incooperation with the bottom wall, the side walls having edges definingan opening leading into the cavity; a protective layer covering theopening and enclosing the cavity; a vision subsystem disposed within thecavity and between the protective layer and the bottom wall and servingto provide a depth map of an object outside the protective layer, thevision subsystem comprising: a clip assembly for carrying opticalcomponents that cooperate to generate information for the depth map, theclip assembly comprising: a first bracket arranged to support and holdthe optical components at a fixed distance from each other and a secondbracket having a body secured to the first bracket, wherein the secondbracket has a projection extending away from the body.

U.S. Pat. No. 9,870,024 B2 describes an electronic display whichincludes several layers, such as a cover layer, a color filter layer, adisplay layer including light emitting diodes or organic light emittingdiodes, a thin film transistor layer, etc. In one embodiment, the layersinclude a substantially transparent region disposed above the camera.The substantially transparent region allows light from outside to reachthe camera, enabling the camera to record an image.

U.S. Pat. No. 10,057,541 B2 describes an image capturing apparatus and aphotographing method. The image capturing apparatus comprises: atransparent display panel; and a camera facing a bottom surface of thetransparent display panel for synchronizing a shutter time with a periodwhen the transparent display panel displays a black image, and forcapturing an image positioned in front of the transparent display panel.

U.S. Pat. No. 10,215,988 B2 describes an optical system for displayinglight from a scene which includes an active optical component thatincludes a first plurality of light directing apertures, an opticaldetector, a processor, a display, and a second plurality of lightdirecting apertures. The first plurality of light directing apertures ispositioned to provide an optical input to the optical detector. Theoptical detector is positioned to receive the optical input and convertthe optical input to an electrical signal corresponding to intensity andlocation data. The processor is connected to receive the data from theoptical detector and process the data for the display. The secondplurality of light directing apertures is positioned to provide anoptical output from the display.

WO 2019/042956 A1 describes a detector for determining a position of atleast one object. The detector comprises —at least one sensor elementhaving a matrix of optical sensors, the optical sensors each having alight-sensitive area, wherein each optical sensor is designed togenerate at least one sensor signal in response to an illumination ofits respective light-sensitive area by a reflection light beampropagating from the object to the detector, wherein the sensor elementis adapted to determine at least one reflection image; —at least oneevaluation device, wherein the evaluation device is adapted to select atleast one reflection feature of the reflection image, wherein theevaluation device is configured for determining at least onelongitudinal region of the selected reflection feature of the reflectionimage by evaluating a combined signal Q from the sensor signals, whereinthe evaluation device is adapted to determine at least one displacementregion in at least one reference image corresponding to the longitudinalregion, wherein the evaluation device is adapted to match the selectedreflection feature with at least one reference feature within thedisplacement region.

Problem Addressed by the Invention

It is therefore an object of the present invention to provide devicesand methods facing the above-mentioned technical challenges of knowndevices and methods. Specifically, it is an object of the presentinvention to provide devices and methods which allow reliable depthmeasurement through a display with a low technical effort and with lowrequirements in terms of technical resources and cost.

SUMMARY OF THE INVENTION

This problem is solved by the invention with the features of theindependent patent claims. Advantageous developments of the invention,which can be realized individually or in combination, are presented inthe dependent claims and/or in the following specification and detailedembodiments.

As used in the following, the terms “have”, “comprise” or “include” orany arbitrary grammatical variations thereof are used in a non-exclusiveway. Thus, these terms may both refer to a situation in which, besidesthe feature introduced by these terms, no further features are presentin the entity described in this context and to a situation in which oneor more further features are present. As an example, the expressions “Ahas B”, “A comprises B” and “A includes B” may both refer to a situationin which, besides B, no other element is present in A (i.e. a situationin which A solely and exclusively consists of B) and to a situation inwhich, besides B, one or more further elements are present in entity A,such as element C, elements C and D or even further elements.

Further, it shall be noted that the terms “at least one”, “one or more”or similar expressions indicating that a feature or element may bepresent once or more than once typically will be used only once whenintroducing the respective feature or element. In the following, in mostcases, when referring to the respective feature or element, theexpressions “at least one” or “one or more” will not be repeated,non-withstanding the fact that the respective feature or element may bepresent once or more than once.

Further, as used in the following, the terms “preferably”, “morepreferably”, “particularly”, “more particularly”, “specifically”, “morespecifically” or similar terms are used in conjunction with optionalfeatures, without restricting alternative possibilities. Thus, featuresintroduced by these terms are optional features and are not intended torestrict the scope of the claims in any way. The invention may, as theskilled person will recognize, be performed by using alternativefeatures. Similarly, features introduced by “in an embodiment of theinvention” or similar expressions are intended to be optional features,without any restriction regarding alternative embodiments of theinvention, without any restrictions regarding the scope of the inventionand without any restriction regarding the possibility of combining thefeatures introduced in such a way with other optional or non-optionalfeatures of the invention.

In a first aspect of the present invention a display device isdisclosed. As used herein, the term “display” may refer to an arbitraryshaped device configured for displaying an item of information such asat least one image, at least one diagram, at least one histogram, atleast one text, at least one sign. The display may be at least onemonitor or at least one screen. The display may have an arbitrary shape,preferably a rectangular shape. As used herein, the term “displaydevice” generally may refer to at least one electronic device comprisingat least one display. For example, the display device may be at leastone device selected from the group consisting of: a television device,smart phones, game consoles, personal computers, laptops, tablets, atleast one virtual reality device, or combinations thereof.

The Display Device Comprises

-   -   at least one illumination source configured for projecting at        least one illumination pattern comprising a plurality of        illumination features on at least one scene;    -   at least one optical sensor having at least one light sensitive        area, wherein the optical sensor is configured for determining        at least one first image comprising a plurality of reflection        features generated by the scene in response to illumination by        the illumination features;    -   at least one translucent display configured for displaying        information, wherein the illumination source and the optical        sensor are placed in direction of propagation of the        illumination pattern in front of the display;    -   at least one evaluation device, wherein the evaluation device is        configured for evaluating the first image, wherein the        evaluation of the first image comprises identifying the        reflection features of the first image and sorting the        identified reflection features with respect to brightness,        wherein each of the reflection features comprises at least one        beam profile, wherein the evaluation device is configured for        determining at least one longitudinal coordinate z_(DPR) for        each of the reflection features by analysis of their beam        profiles,    -   wherein the evaluation device is configured for unambiguously        matching of reflection features with corresponding illumination        features by using the longitudinal coordinate z_(DPR), wherein        the matching is performed with decreasing brightness of the        reflection features starting with the brightest reflection        feature, wherein the evaluation device is configured for        classifying a reflection feature being matched with an        illumination feature as real feature and for classifying a        reflection feature not being matched with an illumination        feature as false feature, wherein the evaluation device is        configured for rejecting the false features and for generating a        depth map for the real features by using the longitudinal        coordinate z_(DPR).

As used herein, the term “scene” may refer to at least one arbitraryobject or spatial region. The scene may comprise the at least one objectand a surrounding environment.

The illumination source is configured for projecting at least oneillumination pattern comprising a plurality of illumination features onthe scene. As used herein, the term “illumination source” may generallyrefers to at least one arbitrary device adapted to provide the at leastone illumination light beam for illumination of the scene. Theillumination source may be adapted to directly or indirectlyilluminating the scene, wherein the illumination pattern is reflected orscattered by surfaces of the scene and, thereby, is at least partiallydirected towards the optical sensor. The illumination source may beadapted to illuminate the scene, for example, by directing a light beamtowards the scene, which reflects the light beam. The illuminationsource may be configured for generating an illuminating light beam forilluminating the scene.

The illumination source may comprise at least one light source. Theillumination source may comprise a plurality of light sources. Theillumination source may comprise an artificial illumination source, inparticular at least one laser source and/or at least one incandescentlamp and/or at least one semiconductor light source, for example, atleast one light-emitting diode, in particular an organic and/orinorganic light-emitting diode. As an example, the light emitted by theillumination source may have a wavelength of 300 to 1100 nm, especially500 to 1100 nm. Additionally or alternatively, light in the infraredspectral range may be used, such as in the range of 780 nm to 3.0 μm.Specifically, the light in the part of the near infrared region wheresilicon photodiodes are applicable specifically in the range of 700 nmto 1100 nm may be used. The illumination source may be configured forgenerating the at least one illumination pattern in the infrared region.Using light in the near infrared region allows that light is not or onlyweakly detected by human eyes and is still detectable by siliconsensors, in particular standard silicon sensors.

As used herein, the term “ray” generally refers to a line that isperpendicular to wavefronts of light which points in a direction ofenergy flow. As used herein, the term “beam” generally refers to acollection of rays. In the following, the terms “ray” and “beam” will beused as synonyms. As further used herein, the term “light beam”generally refers to an amount of light, specifically an amount of lighttraveling essentially in the same direction, including the possibilityof the light beam having a spreading angle or widening angle. The lightbeam may have a spatial extension. Specifically, the light beam may havea non-Gaussian beam profile. The beam profile may be selected from thegroup consisting of a trapezoid beam profile; a triangle beam profile; aconical beam profile. The trapezoid beam profile may have a plateauregion and at least one edge region. The light beam specifically may bea Gaussian light beam or a linear combination of Gaussian light beams,as will be outlined in further detail below. Other embodiments arefeasible, however.

The illumination source may be configured for emitting light at a singlewavelength. Specifically, the wavelength may be in the near infraredregion. In other embodiments, the illumination may be adapted to emitlight with a plurality of wavelengths allowing additional measurementsin other wavelengths channels

The illumination source may be or may comprise at least one multiplebeam light source. For example, the illumination source may comprise atleast one laser source and one or more diffractive optical elements(DOEs). Specifically, the illumination source may comprise at least onelaser and/or laser source. Various types of lasers may be employed, suchas semiconductor lasers, double heterostructure lasers, external cavitylasers, separate confinement heterostructure lasers, quantum cascadelasers, distributed bragg reflector lasers, polariton lasers, hybridsilicon lasers, extended cavity diode lasers, quantum dot lasers, volumeBragg grating lasers, Indium Arsenide lasers, transistor lasers, diodepumped lasers, distributed feedback lasers, quantum well lasers,interband cascade lasers, Gallium Arsenide lasers, semiconductor ringlaser, extended cavity diode lasers, or vertical cavity surface-emittinglasers. Additionally or alternatively, non-laser light sources may beused, such as LEDs and/or light bulbs. The illumination source maycomprise one or more diffractive optical elements (DOEs) adapted togenerate the illumination pattern. For example, the illumination sourcemay be adapted to generate and/or to project a cloud of points, forexample the illumination source may comprise one or more of at least onedigital light processing projector, at least one LCoS projector, atleast one spatial light modulator; at least one diffractive opticalelement; at least one array of light emitting diodes; at least one arrayof laser light sources. On account of their generally defined beamprofiles and other properties of handleability, the use of at least onelaser source as the illumination source is particularly preferred. Theillumination source may be integrated into a housing of the displaydevice.

In one embodiment, the illumination source may be a single or multiplebeam source and may configured for projecting the at least oneillumination pattern such as at least one point pattern. Theillumination pattern may be generated as follows. The illuminationsource may be configured for generating at least one light beam. Theillumination source may be placed in direction of propagation of theillumination pattern in front of the display. Thus, the beam path of thelight beam may pass from the illumination source through the display tothe scene. During its pass through the display the light beam mayexperience diffraction by the display which may result in thecharacteristic illumination pattern such as the point pattern. Thedisplay in this embodiment may function as grating. A wiring of thedisplay, in particular of a screen, may be configured for forming gapsand/or slits and ridges of the grating.

Further, the illumination source may be configured for emittingmodulated or non-modulated light. In case a plurality of illuminationsources is used, the different illumination sources may have differentmodulation frequencies which, as outlined in further detail below, lateron may be used for distinguishing the light beams.

The light beam or light beams generated by the illumination sourcegenerally may propagate parallel to the optical axis or tilted withrespect to the optical axis, e.g. including an angle with the opticalaxis. The display device may be configured such that the light beam orlight beams propagates from the display device towards the scene alongan optical axis of the display device. For this purpose, the displaydevice may comprise at least one reflective element, preferably at leastone prism, for deflecting the illuminating light beam onto the opticalaxis. As an example, the light beam or light beams, such as the laserlight beam, and the optical axis may include an angle of less than 10°,preferably less than 5° or even less than 2°. Other embodiments,however, are feasible. Further, the light beam or light beams may be onthe optical axis or off the optical axis. As an example, the light beamor light beams may be parallel to the optical axis having a distance ofless 10 than 10 mm to the optical axis, preferably less than 5 mm to theoptical axis or even less than 1 mm to the optical axis or may evencoincide with the optical axis.

As used herein, the term “at least one illumination pattern” refers toat least one arbitrary pattern comprising at least one illuminationfeature adapted to illuminate at least one part of the scene. As usedherein, the term “illumination feature” refers to at least one at leastpartially extended feature of the pattern. The illumination pattern maycomprise a single illumination feature. The illumination pattern maycomprise a plurality of illumination features. The illumination patternmay be selected from the group consisting of: at least one pointpattern; at least one line pattern; at least one stripe pattern; atleast one checkerboard pattern; at least one pattern comprising anarrangement of periodic or non periodic features. The illuminationpattern may comprise regular and/or constant and/or periodic patternsuch as a triangular pattern, a rectangular pattern, a hexagonal patternor a pattern comprising further convex tilings. The illumination patternmay exhibit the at least one illumination feature selected from thegroup consisting of: at least one point; at least one line; at least twolines such as parallel or crossing lines; at least one point and oneline; at least one arrangement of periodic or non-periodic feature; atleast one arbitrary shaped featured. The illumination pattern maycomprise at least one pattern selected from the group consisting of: atleast one point pattern, in particular a pseudo-random point pattern; arandom point pattern or a quasi random pattern; at least one Sobolpattern; at least one quasiperiodic pattern; at least one patterncomprising at least one pre-known feature at least one regular pattern;at least one triangular pattern; at least one hexagonal pattern; atleast one rectangular pattern at least one pattern comprising convexuniform tilings; at least one line pattern comprising at least one line;at least one line pattern comprising at least two lines such as parallelor crossing lines. For example, the illumination source may be adaptedto generate and/or to project a cloud of points. The illumination sourcemay comprise the at least one light projector adapted to generate acloud of points such that the illumination pattern may comprise aplurality of point pattern. The illumination source may comprise atleast one mask adapted to generate the illumination pattern from atleast one light beam generated by the illumination source.

A distance between two features of the illumination pattern and/or anarea of the at least one illumination feature may depend on the circleof confusion in the image. As outlined above, the illumination sourcemay comprise the at least one light source configured for generating theat least one illumination pattern. Specifically, the illumination sourcecomprises at least one laser source and/or at least one laser diodewhich is designated for generating laser radiation. The illuminationsource may comprise the at least one diffractive optical element (DOE).The display device may comprise at least one point projector, such asthe at least one laser source and the DOE, adapted to project at leastone periodic point pattern.

As further used herein, the term “projecting at least one illuminationpattern” refers to providing the at least one illumination pattern forilluminating the at least one scene.

For example, the projected illumination pattern may be a periodic pointpattern. The projected illumination pattern may have a low pointdensity. For example, the illumination pattern may comprise at least oneperiodic point pattern having a low point density, wherein theillumination pattern has 2500 points per field of view. In comparisonwith structured light having typically a point density of 10 k-30 k in afield of view of 55×38° the illumination pattern according to thepresent invention may be less dense. This may allow more power per pointsuch that the proposed technique is less dependent on ambient lightcompared to structured light.

The display device may comprise a single camera comprising the opticalsensor. The display device may comprise a plurality of cameras eachcomprising an optical sensor or a plurality of optical sensors.

The optical sensor has at least one light sensitive area. As usedherein, an “optical sensor” generally refers to a light-sensitive devicefor detecting a light beam, such as for detecting an illumination and/ora light spot generated by at least one light beam. As further usedherein, a “light-sensitive area” generally refers to an area of theoptical sensor which may be illuminated externally, by the at least onelight beam, in response to which illumination at least one sensor signalis generated. The light-sensitive area may specifically be located on asurface of the respective optical sensor. Other embodiments, however,are feasible. The display device may comprise a plurality of opticalsensors each having a light sensitive area. As used herein, the term“the optical sensors each having at least one light sensitive area”refers to configurations with a plurality of single optical sensors eachhaving one light sensitive area and to configurations with one combinedoptical sensor having a plurality of light sensitive areas. The term“optical sensor” furthermore refers to a light-sensitive deviceconfigured to generate one output signal. In case the display devicecomprises a plurality of optical sensors, each optical sensor may beembodied such that precisely one light-sensitive area is present in therespective optical sensor, such as by providing precisely onelight-sensitive area which may be illuminated, in response to whichillumination precisely one uniform sensor signal is created for thewhole optical sensor. Thus, each optical sensor may be a single areaoptical sensor. The use of the single area optical sensors, however,renders the setup of the display device specifically simple andefficient. Thus, as an example, commercially available photo-sensors,such as commercially available silicon photodiodes, each havingprecisely one sensitive area, may be used in the set-up. Otherembodiments, however, are feasible.

Preferably, the light sensitive area may be oriented essentiallyperpendicular to an optical axis of the display device. The optical axismay be a straight optical axis or may be bent or even split, such as byusing one or more deflection elements and/or by using one or more beamsplitters, wherein the essentially perpendicular orientation, in thelatter cases, may refer to the local optical axis in the respectivebranch or beam path of the optical set-up.

The optical sensor specifically may be or may comprise at least onephotodetector, preferably inorganic photodetectors, more preferablyinorganic semiconductor photodetectors, most preferably siliconphotodetectors. Specifically, the optical sensor may be sensitive in theinfrared spectral range. All pixels of the matrix or at least a group ofthe optical sensors of the matrix specifically may be identical. Groupsof identical pixels of the matrix specifically may be provided fordifferent spectral ranges, or all pixels may be identical in terms ofspectral sensitivity. Further, the pixels may be identical in sizeand/or with regard to their electronic or optoelectronic properties.Specifically, the optical sensor may be or may comprise at least oneinorganic photodiode which are sensitive in the infrared spectral range,preferably in the range of 700 nm to 3.0 micrometers. Specifically, theoptical sensor may be sensitive in the part of the near infrared regionwhere silicon photodiodes are applicable specifically in the range of700 nm to 1100 nm. Infrared optical sensors which may be used foroptical sensors may be commercially available infrared optical sensors,such as infrared optical sensors commercially available under the brandname Hertzstueck™ from trinamiX™ GmbH, D-67056 Ludwigshafen am Rhein,Germany. Thus, as an example, the optical sensor may comprise at leastone optical sensor of an intrinsic photovoltaic type, more preferably atleast one semiconductor photodiode selected from the group consistingof: a Ge photodiode, an InGaAs photodiode, an extended InGaAsphotodiode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode.Additionally or alternatively, the optical sensor may comprise at leastone optical sensor of an extrinsic photovoltaic type, more preferably atleast one semiconductor photodiode selected from the group consistingof: a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Znphotodiode, a Si:Ga photodiode, a Si:As photodiode. Additionally oralternatively, the optical sensor may comprise at least onephotoconductive sensor such as a PbS or PbSe sensor, a bolometer,preferably a bolometer selected from the group consisting of a VObolometer and an amorphous Si bolometer.

The optical sensor may be sensitive in one or more of the ultraviolet,the visible or the infrared spectral range. Specifically, the opticalsensor may be sensitive in the visible spectral range from 500 nm to 780nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm.Specifically, the optical sensor may be sensitive in the near infraredregion. Specifically, the optical sensor may be sensitive in the part ofthe near infrared region where silicon photodiodes are applicablespecifically in the range of 700 nm to 1000 nm. The optical sensor,specifically, may be sensitive in the infrared spectral range,specifically in the range of 780 nm to 3.0 micrometers. For example, theoptical sensor each, independently, may be or may comprise at least oneelement selected from the group consisting of a photodiode, a photocell,a photoconductor, a phototransistor or any combination thereof. Forexample, the optical sensor may be or may comprise at least one elementselected from the group consisting of a CCD sensor element, a CMOSsensor element, a photodiode, a photocell, a photoconductor, aphototransistor or any combination thereof. Any other type ofphotosensitive element may be used. The photosensitive element generallymay fully or partially be made of inorganic materials and/or may fullyor partially be made of organic materials. Most commonly, one or morephotodiodes may be used, such as commercially available photodiodes,e.g. inorganic semiconductor photodiodes.

The optical sensor may comprise at least one sensor element comprising amatrix of pixels. Thus, as an example, the optical sensor may be part ofor constitute a pixelated optical device. For example, the opticalsensor may be and/or may comprise at least one CCD and/or CMOS device.As an example, the optical sensor may be part of or constitute at leastone CCD and/or CMOS device having a matrix of pixels, each pixel forminga light-sensitive area.

As used herein, the term “sensor element” generally refers to a deviceor a combination of a plurality of devices configured for sensing atleast one parameter. In the present case, the parameter specifically maybe an optical parameter, and the sensor element specifically may be anoptical sensor element. The sensor element may be formed as a unitary,single device or as a combination of several devices. The sensor elementcomprises a matrix of optical sensors. The sensor element may compriseat least one CMOS sensor. The matrix may be composed of independentpixels such as of independent optical sensors. Thus, a matrix ofinorganic photodiodes may be composed. Alternatively, however, acommercially available matrix may be used, such as one or more of a CCDdetector, such as a CCD detector chip, and/or a CMOS detector, such as aCMOS detector chip. Thus, generally, the sensor element may be and/ormay comprise at least one CCD and/or CMOS device and/or the opticalsensors may form a sensor array or may be part of a sensor array, suchas the above-mentioned matrix. Thus, as an example, the sensor elementmay comprise an array of pixels, such as a rectangular array, having mrows and n columns, with m, n, independently, being positive integers.Preferably, more than one column and more than one row is given, i.e.n>1, m>1. Thus, as an example, n may be 2 to 16 or higher and m may be 2to 16 or higher. Preferably, the ratio of the number of rows and thenumber of columns is close to 1. As an example, n and m may be selectedsuch that 0.3 m/n 3, such as by choosing m/n=1:1, 4:3, 16:9 or similar.As an example, the array may be a square array, having an equal numberof rows and columns, such as by choosing m=2, n=2 or m=3, n=3 or thelike.

The matrix may be composed of independent pixels such as of independentoptical sensors. Thus, a matrix of inorganic photodiodes may becomposed. Alternatively, however, a commercially available matrix may beused, such as one or more of a CCD detector, such as a CCD detectorchip, and/or a CMOS detector, such as a CMOS detector chip. Thus,generally, the optical sensor may be and/or may comprise at least oneCCD and/or CMOS device and/or the optical sensors of the display devicemay form a sensor array or may be part of a sensor array, such as theabove-mentioned matrix.

The matrix specifically may be a rectangular matrix having at least onerow, preferably a plurality of rows, and a plurality of columns. As anexample, the rows and columns may be oriented essentially perpendicular.As used herein, the term “essentially perpendicular” refers to thecondition of a perpendicular orientation, with a tolerance of e.g. ±20°or less, preferably a tolerance of ±10° or less, more preferably atolerance of ±5° or less. Similarly, the term “essentially parallel”refers to the condition of a parallel orientation, with a tolerance ofe.g. ±20° or less, preferably a tolerance of ±10° or less, morepreferably a tolerance of ±5° or less. Thus, as an example, tolerancesof less than 20°, specifically less than 10° or even less than 5°, maybe acceptable. In order to provide a wide range of view, the matrixspecifically may have at least 10 rows, preferably at least 500 rows,more preferably at least 1000 rows. Similarly, the matrix may have atleast 10 columns, preferably at least 500 columns, more preferably atleast 1000 columns. The matrix may comprise at least 50 optical sensors,preferably at least 100000 optical sensors, more preferably at least5000000 optical sensors. The matrix may comprise a number of pixels in amulti-mega pixel range. Other embodiments, however, are feasible. Thus,in setups in which an axial rotational symmetry is to be expected,circular arrangements or concentric arrangements of the optical sensorsof the matrix, which may also be referred to as pixels, may bepreferred.

Thus, as an example, the sensor element may be part of or constitute apixelated optical device. For example, the sensor element may be and/ormay comprise at least one CCD and/or CMOS device. As an example, thesensor element may be part of or constitute at least one CCD and/or CMOSdevice having a matrix of pixels, each pixel forming a light-sensitivearea. The sensor element may employ a rolling shutter or global shuttermethod to read out the matrix of optical sensors.

The display device further may comprise at least one transfer device.The display device may further comprise one or more additional elementssuch as one or more additional optical elements. The display device maycomprise at least one optical element selected from the group consistingof: transfer device, such as at least one lens and/or at least one lenssystem, at least one diffractive optical element. The term “transferdevice”, also denoted as “transfer system”, may generally refer to oneor more optical elements which are adapted to modify the light beam,such as by modifying one or more of a beam parameter of the light beam,a width of the light beam or a direction of the light beam. The transferdevice may be adapted to guide the light beam onto the optical sensor.The transfer device specifically may comprise one or more of: at leastone lens, for example at least one lens selected from the groupconsisting of at least one focus-tunable lens, at least one asphericlens, at least one spheric lens, at least one Fresnel lens; at least onediffractive optical element; at least one concave mirror; at least onebeam deflection element, preferably at least one mirror; at least onebeam splitting element, preferably at least one of a beam splitting cubeor a beam splitting mirror; at least one multi-lens system. As usedherein, the term “focal length” of the transfer device refers to adistance over which incident collimated rays which may impinge thetransfer device are brought into a “focus” which may also be denoted as“focal point”. Thus, the focal length constitutes a measure of anability of the transfer device to converge an impinging light beam.Thus, the transfer device may comprise one or more imaging elementswhich can have the effect of a converging lens. By way of example, thetransfer device can have one or more lenses, in particular one or morerefractive lenses, and/or one or more convex mirrors. In this example,the focal length may be defined as a distance from the center of thethin refractive lens to the principal focal points of the thin lens. Fora converging thin refractive lens, such as a convex or biconvex thinlens, the focal length may be considered as being positive and mayprovide the distance at which a beam of collimated light impinging thethin lens as the transfer device may be focused into a single spot.Additionally, the transfer device can comprise at least onewavelength-selective element, for example at least one optical filter.Additionally, the transfer device can be designed to impress apredefined beam profile on the electromagnetic radiation, for example,at the location of the sensor region and in particular the sensor area.The abovementioned optional embodiments of the transfer device can, inprinciple, be realized individually or in any desired combination.

The transfer device may have an optical axis. In particular, the displaydevice and the transfer device have a common optical axis. As usedherein, the term “optical axis of the transfer device” generally refersto an axis of mirror symmetry or rotational symmetry of the lens or lenssystem. The optical axis of the display device may be a line of symmetryof the optical setup of the display device. The display device comprisesat least one transfer device, preferably at least one transfer systemhaving at least one lens. The transfer system, as an example, maycomprise at least one beam path, with the elements of the transfersystem in the beam path being located in a rotationally symmetricalfashion with respect to the optical axis. Still, as will also beoutlined in further detail below, one or more optical elements locatedwithin the beam path may also be off-centered or tilted with respect tothe optical axis. In this case, however, the optical axis may be definedsequentially, such as by interconnecting the centers of the opticalelements in the beam path, e.g. by interconnecting the centers of thelenses, wherein, in this context, the optical sensors are not counted asoptical elements. The optical axis generally may denote the beam path.Therein, the display device may have a single beam path along which alight beam may travel from the object to the optical sensors, or mayhave a plurality of beam paths. As an example, a single beam path may begiven or the beam path may be split into two or more partial beam paths.In the latter case, each partial beam path may have its own opticalaxis. The optical sensors may be located in one and the same beam pathor partial beam path. Alternatively, however, the optical sensors mayalso be located in different partial beam paths.

The transfer device may constitute a coordinate system, wherein alongitudinal coordinate is a coordinate along the optical axis andwherein d is a spatial offset from the optical axis. The coordinatesystem may be a polar coordinate system in which the optical axis of thetransfer device forms a z-axis and in which a distance from the z-axisand a polar angle may be used as additional coordinates. A directionparallel or antiparallel to the z-axis may be considered a longitudinaldirection, and a coordinate along the z-axis may be considered alongitudinal coordinate. Any direction perpendicular to the z-axis maybe considered a transversal direction, and the polar coordinate and/orthe polar angle may be considered a transversal coordinate.

The display device may constitute a coordinate system in which anoptical axis of the display device forms the z-axis and in which,additionally, an x-axis and a y-axis may be provided which areperpendicular to the z-axis and which are perpendicular to each other.As an example, the display device and/or a part of the display devicemay rest at a specific point in this coordinate system, such as at theorigin of this coordinate system. In this coordinate system, a directionparallel or antiparallel to the z-axis may be regarded as a longitudinaldirection, and a coordinate along the z-axis may be considered alongitudinal coordinate. An arbitrary direction perpendicular to thelongitudinal direction may be considered a transversal direction, and anx-and/or y-coordinate may be considered a transversal coordinate.

Alternatively, other types of coordinate systems may be used. Thus, asan example, a polar coordinate system may be used in which the opticalaxis forms a z-axis and in which a distance from the z-axis and a polarangle may be used as additional coordinates. Again, a direction parallelor antiparallel to the z-axis may be considered a longitudinaldirection, and a coordinate along the z-axis may be considered alongitudinal coordinate. Any direction perpendicular to the z-axis maybe considered a transversal direction, and the polar coordinate and/orthe polar angle may be considered a transversal coordinate.

The optical sensor is configured for determining at least one firstimage comprising a plurality of reflection features generated by thescene in response to illumination by the illumination features. As usedherein, without limitation, the term “image” specifically may relate todata recorded by using the optical sensor, such as a plurality ofelectronic readings from an imaging device, such as the pixels of thesensor element. The image itself, thus, may comprise pixels, the pixelsof the image correlating to pixels of the matrix of the sensor element.Consequently, when referring to “pixels”, reference is either made tothe units of image information generated by the single pixels of thesensor element or to the single pixels of the sensor element directly.As used herein, the term “two dimensional image” may generally refer toan image having information about transversal coordinates such as thedimensions of height and width only. As used herein, the term “threedimensional image” may generally refer to an image having informationabout transversal coordinates and additionally about the longitudinalcoordinate such as the dimensions of height, width and depth. As usedherein, the term “reflection feature” may refer to a feature in an imageplane generated by the scene in response to illumination, specificallywith at least one illumination feature.

The display device comprises the at least one translucent displayconfigured for displaying information. As used herein, the term“translucent” may refer to a property of the display to allow light, inparticular of a certain wavelength range, to pass through. Theillumination source and the optical sensor are placed in direction ofpropagation of the illumination pattern in front of the display. Theillumination source and the optical sensor may be arranged in a fixedposition with respect to each other. For example, the setup of thedisplay device may comprise a camera, comprising the optical sensor anda lens system, and a laser projector. The laser projector and the cameramay be fixed, in a direction of propagation of light reflected by thescene, behind the translucent display. The laser projector may generatea dot pattern and shines through the display. The camera may lookthrough the display. The arrangement of the illumination source andoptical sensor in a direction of propagation of light reflected by thescene, behind the translucent display, however, may result in thatdiffraction grating of the display generates multiple laser points onthe scene and also in the first image. Thereby these multiple spots onthe first image may not include any useful distance information. As willbe outlined in detail below, the evaluation device may be configured forfinding and evaluating the reflection features of the zero order ofdiffraction grating, i.e. real features, and may neglect the reflectionfeatures of the higher orders, i.e. false features.

The display device comprises the at least one evaluation device. Theevaluation device is configured for evaluating the first image. Asfurther used herein, the term “evaluation device” generally refers to anarbitrary device adapted to perform the named operations, preferably byusing at least one data processing device and, more preferably, by usingat least one processor and/or at least one application-specificintegrated circuit. Thus, as an example, the at least one evaluationdevice may comprise at least one data processing device having asoftware code stored thereon comprising a number of computer commands.The evaluation device may provide one or more hardware elements forperforming one or more of the named operations and/or may provide one ormore processors with software running thereon for performing one or moreof the named operations. Operations, including evaluating the images.Specifically the determining the beam profile and indication of thesurface, may be performed by the at least one evaluation device. Thus,as an example, one or more instructions may be implemented in softwareand/or hardware. Thus, as an example, the evaluation device may compriseone or more programmable devices such as one or more computers,application-specific integrated circuits (ASICs), Digital SignalProcessors (DSPs), or Field Programmable Gate Arrays (FPGAs) which areconfigured to perform the above-mentioned evaluation. Additionally oralternatively, however, the evaluation device may also fully orpartially be embodied by hardware.

The evaluation device and the display device may fully or partially beintegrated into a single device. Thus, generally, the evaluation devicealso may form part of the display device. Alternatively, the evaluationdevice and the display device may fully or partially be embodied asseparate devices. The display device may comprise further components.

The evaluation device may be or may comprise one or more integratedcircuits, such as one or more application-specific integrated circuits(ASICs), and/or one or more data processing devices, such as one or morecomputers, preferably one or more microcomputers and/ormicrocontrollers, Field Programmable Arrays, or Digital SignalProcessors. Additional components may be comprised, such as one or morepreprocessing devices and/or data acquisition devices, such as one ormore devices for receiving and/or preprocessing of the sensor signals,such as one or more AD-converters and/or one or more filters. Further,the evaluation device may comprise one or more measurement devices, suchas one or more measurement devices for measuring electrical currentsand/or electrical voltages. Further, the evaluation device may compriseone or more data storage devices. Further, the evaluation device maycomprise one or more interfaces, such as one or more wireless interfacesand/or one or more wire-bound interfaces.

The evaluation device can be connected to or may comprise at least onefurther data processing device that may be used for one or more ofdisplaying, visualizing, analyzing, distributing, communicating orfurther processing of information, such as information obtained by theoptical sensor and/or by the evaluation device. The data processingdevice, as an example, may be connected or incorporate at least one of adisplay, a projector, a monitor, an LCD, a TFT, a loudspeaker, amultichannel sound system, an LED pattern, or a further visualizationdevice. It may further be connected or incorporate at least one of acommunication device or communication interface, a connector or a port,capable of sending encrypted or unencrypted information using one ormore of email, text messages, telephone, Bluetooth, Wi-Fi, infrared orinternet interfaces, ports or connections. It may further be connectedto or incorporate at least one of a processor, a graphics processor, aCPU, an Open Multimedia Applications Platform (OMAP™), an integratedcircuit, a system on a chip such as products from the Apple A series orthe Samsung S3C2 series, a microcontroller or microprocessor, one ormore memory blocks such as ROM, RAM, EEPROM, or flash memory, timingsources such as oscillators or phase-locked loops, counter-timers,real-time timers, or power-on reset generators, voltage regulators,power management circuits, or DMA controllers. Individual units mayfurther be connected by buses such as AMBA buses or be integrated in anInternet of Things or Industry 4.0 type network.

The evaluation device and/or the data processing device may be connectedby or have further external interfaces or ports such as one or more ofserial or parallel interfaces or ports, USB, Centronics Port, FireWire,HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analogueinterfaces or ports such as one or more of ADCs or DACs, or standardizedinterfaces or ports to further devices such as a 2D-camera device usingan RGB-interface such as CameraLink. The evaluation device and/or thedata processing device may further be connected by one or more ofinterprocessor interfaces or ports, FPGA-FPGA-interfaces, or serial orparallel interfaces ports. The evaluation device and the data processingdevice may further be connected to one or more of an optical disc drive,a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a diskdrive, a hard disk drive, a solid state disk or a solid state hard disk.

The evaluation device and/or the data processing device may be connectedby or have one or more further external connectors such as one or moreof phone connectors, RCA connectors, VGA connectors, hermaphroditeconnectors, USB connectors, HDMI connectors, 8P8C connectors, BCNconnectors, IEC 60320 C14 connectors, optical fiber connectors,D-subminiature connectors, RF connectors, coaxial connectors, SCARTconnectors, XLR connectors, and/or may incorporate at least one suitablesocket for one or more of these connectors.

The evaluation device is configured for evaluating of the first image.The evaluation of the first image comprises identifying the reflectionfeatures of the first image. The evaluation device may be configured forperforming at least one image analysis and/or image processing in orderto identify the reflection features. The image analysis and/or imageprocessing may use at least one feature detection algorithm. The imageanalysis and/or image processing may comprise one or more of thefollowing: a filtering; a selection of at least one region of interest;a formation of a difference image between an image created by the sensorsignals and at least one offset; an inversion of sensor signals byinverting an image created by the sensor signals; a formation of adifference image between an image created by the sensor signals atdifferent times; a background correction; a decomposition into colorchannels; a decomposition into hue; saturation; and brightness channels;a frequency decomposition; a singular value decomposition; applying ablob detector; applying a corner detector; applying a Determinant ofHessian filter; applying a principle curvature-based region detector;applying a maximally stable extremal regions detector; applying ageneralized Hough-transformation; applying a ridge detector; applying anaffine invariant feature detector; applying an affine-adapted interestpoint operator; applying a Harris affine region detector; applying aHessian affine region detector; applying a scale-invariant featuretransform; applying a scale-space extrema detector; applying a localfeature detector; applying speeded up robust features algorithm;applying a gradient location and orientation histogram algorithm;applying a histogram of oriented gradients descriptor; applying aDeriche edge detector; applying a differential edge detector; applying aspatio-temporal interest point detector; applying a Moravec cornerdetector; applying a Canny edge detector; applying a Laplacian ofGaussian filter; applying a Difference of Gaussian filter; applying aSobel operator; applying a Laplace operator; applying a Scharr operator;applying a Prewitt operator; applying a Roberts operator; applying aKirsch operator; applying a high-pass filter; applying a low-passfilter; applying a Fourier transformation; applying aRadon-transformation; applying a Hough-transformation; applying awavelet-transformation; a thresholding; creating a binary image. Theregion of interest may be determined manually by a user or may bedetermined automatically, such as by recognizing a feature within theimage generated by the optical sensor.

For example, the illumination source may be configured for generatingand/or projecting a cloud of points such that a plurality of illuminatedregions is generated on the optical sensor, for example the CMOSdetector. Additionally, disturbances may be present on the opticalsensor such as disturbances due to speckles and/or extraneous lightand/or multiple reflections. The evaluation device may be adapted todetermine at least one region of interest, for example one or morepixels illuminated by the light beam which are used for determination ofthe longitudinal coordinate of the object. For example, the evaluationdevice may be adapted to perform a filtering method, for example, ablob-analysis and/or an edge filter and/or object recognition method.

The evaluation device may be configured for performing at least oneimage correction. The image correction may comprise at least onebackground subtraction. The evaluation device may be adapted to removeinfluences from background light from the beam profile, for example, byan imaging without further illumination.

Each of the reflection features comprises at least one beam profile. Asused herein, the term “beam profile” of the reflection feature maygenerally refer to at least one intensity distribution of the reflectionfeature, such as of a light spot on the optical sensor, as a function ofthe pixel. The beam profile may be selected from the group consisting ofa trapezoid beam profile; a triangle beam profile; a conical beamprofile and a linear combination of Gaussian beam profiles. Theevaluation device is configured for determining beam profile informationfor each of the reflection features by analysis of their beam profiles.

The evaluation device is configured for determining at least onelongitudinal coordinate z_(DPR) for each of the reflection features byanalysis of their beam profiles. As used herein, the term “analysis ofthe beam profile” may generally refer to evaluating of the beam profileand may comprise at least one mathematical operation and/or at least onecomparison and/or at least symmetrizing and/or at least one filteringand/or at least one normalizing. For example, the analysis of the beamprofile may comprise at least one of a histogram analysis step, acalculation of a difference measure, application of a neural network,application of a machine learning algorithm. The evaluation device maybe configured for symmetrizing and/or for normalizing and/or forfiltering the beam profile, in particular to remove noise or asymmetriesfrom recording under larger angles, recording edges or the like. Theevaluation device may filter the beam profile by removing high spatialfrequencies such as by spatial frequency analysis and/or medianfiltering or the like. Summarization may be performed by center ofintensity of the light spot and averaging all intensities at the samedistance to the center. The evaluation device may be configured fornormalizing the beam profile to a maximum intensity, in particular toaccount for intensity differences due to the recorded distance. Theevaluation device may be configured for removing influences frombackground light from the beam profile, for example, by an imagingwithout illumination.

The reflection feature may cover or may extend over at least one pixelof the image. For example, the reflection feature may cover or mayextend over plurality of pixels. The evaluation device may be configuredfor determining and/or for selecting all pixels connected to and/orbelonging to the reflection feature, e.g. a light spot. The evaluationdevice may be configured for determining the center of intensity by

${R_{coi} = \frac{1}{l \cdot {\sum{j \cdot r_{pixel}}}}},$

wherein R_(coi), is a position of center of intensity, r_(pixel) is thepixel position and l=Σ_(j)I_(total) with being the number of pixels Jconnected to and/or belonging to the reflection feature and I_(total)being the total intensity.

The evaluation device may be configured for determining the longitudinalcoordinate z_(DPR) for each of the reflection features by usingdepth-from-photon-ratio technique, also denoted beam profile analysis.With respect to depth-from-photon-ratio (DPR) technique reference ismade to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, thefull content of which is included by reference.

The evaluation device may be configured for determining the beam profileof each of the reflection features. As used herein, the term“determining the beam profile” refers to identifying at least onereflection feature provided by the optical sensor and/or selecting atleast one reflection feature provided by the optical sensor andevaluating at least one intensity distribution of the reflectionfeature. As an example, a region of the matrix may be used and evaluatedfor determining the intensity distribution, such as a three-dimensionalintensity distribution or a two-dimensional intensity distribution, suchas along an axis or line through the matrix. As an example, a center ofillumination by the light beam may be determined, such as by determiningthe at least one pixel having the highest illumination, and across-sectional axis may be chosen through the center of illumination.The intensity distribution may an intensity distribution as a functionof a coordinate along this cross-sectional axis through the center ofillumination. Other evaluation algorithms are feasible.

The analysis of the beam profile of one of the reflection features maycomprise determining at least one first area and at least one secondarea of the beam profile. The first area of the beam profile may be anarea A1 and the second area of the beam profile may be an area A2. Theevaluation device may be configured for integrating the first area andthe second area. The evaluation device may be configured to derive acombined signal, in particular a quotient Q, by one or more of dividingthe integrated first area and the integrated second area, dividingmultiples of the integrated first area and the integrated second area,dividing linear combinations of the integrated first area and theintegrated second area. The evaluation device may configured fordetermining at least two areas of the beam profile and/or to segment thebeam profile in at least two segments comprising different areas of thebeam profile, wherein overlapping of the areas may be possible as longas the areas are not congruent. For example, the evaluation device maybe configured for determining a plurality of areas such as two, three,four, five, or up to ten areas. The evaluation device may be configuredfor segmenting the light spot into at least two areas of the beamprofile and/or to segment the beam profile in at least two segmentscomprising different areas of the beam profile. The evaluation devicemay be configured for determining for at least two of the areas anintegral of the beam profile over the respective area. The evaluationdevice may be configured for comparing at least two of the determinedintegrals. Specifically, the evaluation device may be configured fordetermining at least one first area and at least one second area of thebeam profile. As used herein, the term “area of the beam profile”generally refers to an arbitrary region of the beam profile at theposition of the optical sensor used for determining the quotient Q. Thefirst area of the beam profile and the second area of the beam profilemay be one or both of adjacent or overlapping regions. The first area ofthe beam profile and the second area of the beam profile may be notcongruent in area. For example, the evaluation device may be configuredfor dividing a sensor region of the CMOS sensor into at least twosub-regions, wherein the evaluation device may be configured fordividing the sensor region of the CMOS sensor into at least one leftpart and at least one right part and/or at least one upper part and atleast one lower part and/or at least one inner and at least one outerpart. Additionally or alternatively, the display device may comprise atleast two optical sensors, wherein the light-sensitive areas of a firstoptical sensor and of a second optical sensor may be arranged such thatthe first optical sensor is adapted to determine the first area of thebeam profile of the reflection feature and that the second opticalsensor is adapted to determine the second area of the beam profile ofthe reflection feature. The evaluation device may be adapted tointegrate the first area and the second area. The evaluation device maybe configured for using at least one predetermined relationship betweenthe quotient Q and the longitudinal coordinate for determining thelongitudinal coordinate. The predetermined relationship may be one ormore of an empiric relationship, a semi-empiric relationship and ananalytically derived relationship. The evaluation device may comprise atleast one data storage device for storing the pre-determinedrelationship, such as a lookup list or a lookup table.

The first area of the beam profile may comprise essentially edgeinformation of the beam profile and the second area of the beam profilecomprises essentially center information of the beam profile, and/or thefirst area of the beam profile may comprise essentially informationabout a left part of the beam profile and the second area of the beamprofile comprises essentially information about a right part of the beamprofile. The beam profile may have a center, i.e. a maximum value of thebeam profile and/or a center point of a plateau of the beam profileand/or a geometrical center of the light spot, and falling edgesextending from the center. The second region may comprise inner regionsof the cross section and the first region may comprise outer regions ofthe cross section. As used herein, the term “essentially centerinformation” generally refers to a low proportion of edge information,i.e. proportion of the intensity distribution corresponding to edges,compared to a proportion of the center information, i.e. proportion ofthe intensity distribution corresponding to the center. Preferably, thecenter information has a proportion of edge information of less than10%, more preferably of less than 5%, most preferably the centerinformation comprises no edge content. As used herein, the term“essentially edge information” generally refers to a low proportion ofcenter information compared to a proportion of the edge information. Theedge information may comprise information of the whole beam profile, inparticular from center and edge regions. The edge information may have aproportion of center information of less than 10%, preferably of lessthan 5%, more preferably the edge information comprises no centercontent. At least one area of the beam profile may be determined and/orselected as second area of the beam profile if it is close or around thecenter and comprises essentially center information. At least one areaof the beam profile may be determined and/or selected as first area ofthe beam profile if it comprises at least parts of the falling edges ofthe cross section. For example, the whole area of the cross section maybe determined as first region.

Other selections of the first area A1 and second area A2 may befeasible. For example, the first area may comprise essentially outerregions of the beam profile and the second area may comprise essentiallyinner regions of the beam profile. For example, in case of atwo-dimensional beam profile, the beam profile may be divided in a leftpart and a right part, wherein the first area may comprise essentiallyareas of the left part of the beam profile and the second area maycomprise essentially areas of the right part of the beam profile.

The edge information may comprise information relating to a number ofphotons in the first area of the beam profile and the center informationmay comprise information relating to a number of photons in the secondarea of the beam profile. The evaluation device may be configured fordetermining an area integral of the beam profile. The evaluation devicemay be configured for determining the edge information by integratingand/or summing of the first area. The evaluation device may beconfigured for determining the center information by integrating and/orsumming of the second area. For example, the beam profile may be atrapezoid beam profile and the evaluation device may be configured fordetermining an integral of the trapezoid. Further, when trapezoid beamprofiles may be assumed, the determination of edge and center signalsmay be replaced by equivalent evaluations making use of properties ofthe trapezoid beam profile such as determination of the slope andposition of the edges and of the height of the central plateau andderiving edge and center signals by geometric considerations.

In one embodiment, A1 may correspond to a full or complete area of afeature point on the optical sensor. A2 may be a central area of thefeature point on the optical sensor. The central area may be a constantvalue. The central area may be smaller compared to the full area of thefeature point. For example, in case of a circular feature point, thecentral area may have a radius from 0.1 to 0.9 of a full radius of thefeature point, preferably from 0.4 to 0.6 of the full radius.

In one embodiment, the illumination pattern may comprise at least oneline pattern. A1 may correspond to an area with a full line width of theline pattern on the optical sensors, in particular on the lightsensitive area of the optical sensors. The line pattern on the opticalsensor may be widened and/or displaced compared to the line pattern ofthe illumination pattern such that the line width on the optical sensorsis increased. In particular, in case of a matrix of optical sensors, theline width of the line pattern on the optical sensors may change fromone column to another column. A2 may be a central area of the linepattern on the optical sensor. The line width of the central area may bea constant value, and may in particular correspond to the line width inthe illumination pattern. The central area may have a smaller line widthcompared to the full line width. For example, the central area may havea line width from 0.1 to 0.9 of the full line width, preferably from 0.4to 0.6 of the full line width. The line pattern may be segmented on theoptical sensors. Each column of the matrix of optical sensors maycomprise center information of intensity in the central area of the linepattern and edge information of intensity from regions extending furtheroutwards from the central area to edge regions of the line pattern.

In one embodiment, the illumination pattern may comprise at least pointpattern. A1 may correspond to an area with a full radius of a point ofthe point pattern on the optical sensors. A2 may be a central area ofthe point in the point pattern on the optical sensors. The central areamay be a constant value. The central area may have a radius compared tothe full radius. For example, the central area may have a radius from0.1 to 0.9 of the full radius, preferably from 0.4 to 0.6 of the fullradius.

The illumination pattern may comprise both at least one point patternand at least one line pattern. Other embodiments in addition oralternatively to line pattern and point pattern are feasible.

The evaluation device may be configured to derive the quotient Q by oneor more of dividing the first area and the second area, dividingmultiples of the first area and the second area, dividing linearcombinations of the first area and the second area. The evaluationdevice may be configured for deriving the quotient Q by

$Q = \frac{\int{\int_{A1}{{E\left( {x,y} \right)}dxdy}}}{\int{\int_{A2}{{E\left( {x,y} \right)}dxdy}}}$

wherein x and y are transversal coordinates, A1 and A2 are the first andsecond area of the beam profile, respectively, and E(x,y) denotes thebeam profile.

Additionally or alternatively, the evaluation device may be adapted todetermine one or both of center information or edge information from atleast one slice or cut of the light spot. This may be realized, forexample, by replacing the area integrals in the quotient Q by a lineintegral along the slice or cut. For improved accuracy, several slicesor cuts through the light spot may be used and averaged. In case of anelliptical spot profile, averaging over several slices or cuts mayresult in improved distance information.

For example, in case of the optical sensor having a matrix of pixels,the evaluation device may be configured for evaluating the beam profile,by

-   -   determining the pixel having the highest sensor signal and        forming at least one center signal;    -   evaluating sensor signals of the matrix and forming at least one        sum signal;    -   determining the quotient Q by combining the center signal and        the sum signal; and    -   determining at least one longitudinal coordinate z of the object        by evaluating the quotient Q.

As used herein, a “sensor signal” generally refers to a signal generatedby the optical sensor and/or at least one pixel of the optical sensor inresponse to illumination. Specifically, the sensor signal may be or maycomprise at least one electrical signal, such as at least one analogueelectrical signal and/or at least one digital electrical signal. Morespecifically, the sensor signal may be or may comprise at least onevoltage signal and/or at least one current signal. More specifically,the sensor signal may comprise at least one photocurrent. Further,either raw sensor signals may be used, or the display device, theoptical sensor or any other element may be adapted to process orpreprocess the sensor signal, thereby generating secondary sensorsignals, which may also be used as sensor signals, such as preprocessingby filtering or the like. The term “center signal” generally refers tothe at least one sensor signal comprising essentially center informationof the beam profile. As used herein, the term “highest sensor signal”refers to one or both of a local maximum or a maximum in a region ofinterest. For example, the center signal may be the signal of the pixelhaving the highest sensor signal out of the plurality of sensor signalsgenerated by the pixels of the entire matrix or of a region of interestwithin the matrix, wherein the region of interest may be predeterminedor determinable within an image generated by the pixels of the matrix.The center signal may arise from a single pixel or from a group ofoptical sensors, wherein, in the latter case, as an example, the sensorsignals of the group of pixels may be added up, integrated or averaged,in order to determine the center signal. The group of pixels from whichthe center signal arises may be a group of neighboring pixels, such aspixels having less than a predetermined distance from the actual pixelhaving the highest sensor signal, or may be a group of pixels generatingsensor signals being within a pre-determined range from the highestsensor signal. The group of pixels from which the center signal arisesmay be chosen as large as possible in order to allow maximum dynamicrange. The evaluation device may be adapted to determine the centersignal by integration of the plurality of sensor signals, for examplethe plurality of pixels around the pixel having the highest sensorsignal. For example, the beam profile may be a trapezoid beam profileand the evaluation device may be adapted to determine an integral of thetrapezoid, in particular of a plateau of the trapezoid.

As outlined above, the center signal generally may be a single sensorsignal, such as a sensor signal from the pixel in the center of thelight spot, or may be a combination of a plurality of sensor signals,such as a combination of sensor signals arising from pixels in thecenter of the light spot, or a secondary sensor signal derived byprocessing a sensor signal derived by one or more of the aforementionedpossibilities. The determination of the center signal may be performedelectronically, since a comparison of sensor signals is fairly simplyimplemented by conventional electronics, or may be performed fully orpartially by software. Specifically, the center signal may be selectedfrom the group consisting of: the highest sensor signal; an average of agroup of sensor signals being within a predetermined range of tolerancefrom the highest sensor signal; an average of sensor signals from agroup of pixels containing the pixel having the highest sensor signaland a predetermined group of neighboring pixels; a sum of sensor signalsfrom a group of pixels containing the pixel having the highest sensorsignal and a pre-determined group of neighboring pixels; a sum of agroup of sensor signals being within a pre-determined range of tolerancefrom the highest sensor signal; an average of a group of sensor signalsbeing above a predetermined threshold; a sum of a group of sensorsignals being above a predetermined threshold; an integral of sensorsignals from a group of optical sensors containing the optical sensorhaving the highest sensor signal and a predetermined group ofneighboring pixels; an integral of a group of sensor signals beingwithin a predetermined range of tolerance from the highest sensorsignal; an integral of a group of sensor signals being above apre-determined threshold.

Similarly, the term “sum signal” generally refers to a signal comprisingessentially edge information of the beam profile. For example, the sumsignal may be derived by adding up the sensor signals, integrating overthe sensor signals or averaging over the sensor signals of the entirematrix or of a region of interest within the matrix, wherein the regionof interest may be pre-determined or determinable within an imagegenerated by the optical sensors of the matrix. When adding up,integrating over or averaging over the sensor signals, the actualoptical sensors from which the sensor signal is generated may be leftout of the adding, integration or averaging or, alternatively, may beincluded into the adding, integration or averaging. The evaluationdevice may be adapted to determine the sum signal by integrating signalsof the entire matrix, or of the region of interest within the matrix.For example, the beam profile may be a trapezoid beam profile and theevaluation device may be adapted to determine an integral of the entiretrapezoid. Further, when trapezoid beam profiles may be assumed, thedetermination of edge and center signals may be replaced by equivalentevaluations making use of properties of the trapezoid beam profile suchas determination of the slope and position of the edges and of theheight of the central plateau and deriving edge and center signals bygeometric considerations.

Similarly, the center signal and edge signal may also be determined byusing segments of the beam profile such as circular segments of the beamprofile. For example, the beam profile may be divided into two segmentsby a secant or a chord that does not pass the center of the beamprofile. Thus, one segment will essentially contain edge information,while the other segment will contain essentially center information. Forexample, to further reduce the amount of edge information in the centersignal, the edge signal may further be subtracted from the centersignal.

The quotient Q may be a signal which is generated by combining thecenter signal and the sum signal. Specifically, the determining mayinclude one or more of: forming a quotient of the center signal and thesum signal or vice versa; forming a quotient of a multiple of the centersignal and a multiple of the sum signal or vice versa; forming aquotient of a linear combination of the center signal and a linearcombination of the sum signal or vice versa. Additionally oralternatively, the quotient Q may comprise an arbitrary signal or signalcombination which contains at least one item of information on acomparison between the center signal and the sum signal.

As used herein, the term “longitudinal coordinate of the object” refersto a distance between the optical sensor and the object. The evaluationdevice may be configured for using the at least one predeterminedrelationship between the quotient Q and the longitudinal coordinate fordetermining the longitudinal coordinate. The predetermined relationshipmay be one or more of an empiric relationship, a semi-empiricrelationship and an analytically derived relationship. The evaluationdevice may comprise at least one data storage device for storing thepre-determined relationship, such as a lookup list or a lookup table.

The evaluation device may be configured for executing at least onedepth-from-photon-ratio algorithm which computes distances for allreflection features with zero order and higher order.

The evaluation of the first image comprises sorting the identifiedreflection features with respect to brightness. As used herein, the term“sorting” may refer to assigning a sequence of the reflection featuresfor further evaluation with respect to brightness, in particularstarting with the reflection feature having maximum brightness andsubsequent the reflection features with decreasing brightness. As usedherein, the term “brightness” may refer to magnitude of the reflectionfeature in the first image and/or intensity of the reflection feature inthe first image. The brightness may refer to a defined passband, such asin the visible or infrared spectral range, or may be wavelengthsindependent. The sorting with decreasing brightness may refer to sortingaccording to decreasing brightness and/or sorting with respect todecreasing brightness. The robustness of the determining of thelongitudinal coordinate z_(DPR) can be increased if the brightestreflection features are preferred for DPR computation. This is mainlybecause reflection features with zero order of diffraction grating arealways brighter than false features with a higher order.

The evaluation device is configured for unambiguously matching ofreflection features with corresponding illumination features by usingthe longitudinal coordinate z_(DPR). The longitudinal coordinatedetermined with the depth-from-photon-ratio technique can be used forsolving the so called correspondence problem. In that way, distanceinformation per reflection feature can be used to find thecorrespondence of the known laser projector grid. As used herein, theterm “matching” refers to identifying and/or determining and/orevaluating the corresponding illumination features and reflectionfeatures. As used herein, the term “corresponding illumination featuresand reflection features” may refer to the fact that each of theillumination features of the illumination pattern generates at the scenea reflection feature, wherein the generated reflection feature isassigned to the illumination feature having generated said reflectionfeature. As used herein, the term “unambiguously matching” may refer tothat only one reflection feature is assigned to one illumination featureand/or that no other reflection features can be assigned to the samematched illumination feature.

The illumination feature corresponding to the reflection feature may bedetermined using epipolar geometry. For description of epipolar geometryreference is made, for example, to chapter 2 in X. Jiang, H. Bunke:Dreidimensionales Computersehen” Springer, Berlin Heidelberg, 1997.Epipolar geometry may assume that an illumination image, i.e. an imageof the non-distorted illumination pattern, and the first image may beimages determined at different spatial positions and/or spatialorientations having a fixed distance. The distance may be a relativedistance, also denoted as baseline. The illumination image may be alsodenoted as reference image. The evaluation device may be adapted todetermine an epipolar line in the reference image. The relative positionof the reference image and first image may be known. For example, therelative position of the reference image and the first image may bestored within at least one storage unit of the evaluation device. Theevaluation device may be adapted to determine a straight line extendingfrom a selected reflection feature of the first image to a real worldfeature from which it originates. Thus, the straight line may comprisepossible object features corresponding to the selected reflectionfeature. The straight line and the baseline span an epipolar plane. Asthe reference image is determined at a different relative constellationfrom the first image, the corresponding possible object features may beimaged on a straight line, called epipolar line, in the reference image.The epipolar line may be the intersection of the epipolar plane and thereference image. Thus, a feature of the reference image corresponding tothe selected feature of the first image lies on the epipolar line.

Depending on the distance to the object of the scene having reflectedthe illumination feature, the reflection feature corresponding to theillumination feature may be displaced within the first image. Thereference image may comprise at least one displacement region in whichthe illumination feature corresponding to the selected reflectionfeature would be imaged. The displacement region may comprise only oneillumination feature. The displacement region may also comprise morethan one illumination feature. The displacement region may comprise anepipolar line or a section of an epipolar line. The displacement regionmay comprise more than one epipolar line or more sections of more thanone epipolar line. The displacement region may extend along the epipolarline, orthogonal to an epipolar line, or both. The evaluation device maybe adapted to determine the illumination feature along the epipolarline. The evaluation device may be adapted to determine the longitudinalcoordinate z for the reflection feature and an error interval ±£ fromthe combined signal Q to determine a displacement region along anepipolar line corresponding to Z±ε or orthogonal to an epipolar line.The measurement uncertainty of the distance measurement using thecombined signal Q may result in a displacement region in the secondimage which is non-circular since the measurement uncertainty may bedifferent for different directions. Specifically, the measurementuncertainty along the epipolar line or epipolar lines may be greaterthan the measurement uncertainty in an orthogonal direction with respectto the epipolar line or lines. The displacement region may comprise anextend in an orthogonal direction with respect to the epipolar line orepipolar lines. The evaluation device may be adapted to match theselected reflection feature with at least one illumination featurewithin the displacement region. The evaluation device may be adapted tomatch the selected feature of the first image with the illuminationfeature within the displacement region by using at least one evaluationalgorithm considering the determined longitudinal coordinate z_(DPR).The evaluation algorithm may be a linear scaling algorithm. Theevaluation device may be adapted to determine the epipolar line closestto and/or within the displacement region. The evaluation device may beadapted to determine the epipolar line closest to the image position ofthe reflection feature. The extent of the displacement region along theepipolar line may be larger than the extent of the displacement regionorthogonal to the epipolar line. The evaluation device may be adapted todetermine an epipolar line before determining a correspondingillumination feature. The evaluation device may determine a displacementregion around the image position of each reflection feature. Theevaluation device may be adapted to assign an epipolar line to eachdisplacement region of each image position of the reflection features,such as by assigning the epipolar line closest to a displacement regionand/or within a displacement region and/or closest to a displacementregion along a direction orthogonal to the epipolar line. The evaluationdevice may be adapted to determine the illumination featurecorresponding to the reflection feature by determining the illuminationfeature closest to the assigned displacement region and/or within theassigned displacement region and/or closest to the assigned displacementregion along the assigned epipolar line and/or within the assigneddisplacement region along the assigned epipolar line.

Additionally or alternatively, the evaluation device may be configuredto perform the following steps:

-   -   Determining a displacement region for the image position of each        reflection feature;    -   Assigning an epipolar line to the displacement region of each        reflection feature such as by assigning the epipolar line        closest to a displacement region and/or within a displacement        region and/or closest to a displacement region along a direction        orthogonal to the epipolar line;    -   Assigning and/or determining at least one illumination feature        to each reflection feature such as by assigning the illumination        feature closest to the assigned displacement region and/or        within the assigned displacement region and/or closest to the        assigned displacement region along the assigned epipolar line        and/or within the assigned displacement region along the        assigned epipolar line.

Additionally or alternatively, the evaluation device may be adapted todecide between more than one epipolar line and/or illumination featureto be assigned to a reflection feature such as by comparing distances ofreflection features and/or epipolar lines within the illumination imageand/or by comparing error weighted distances, such as 6-weighteddistances of illumination features and/or epipolar lines within theillumination image and assigning the epipolar line and/or illuminationfeature in shorter distance and/or 6-weighted distance to theillumination feature and/or reflection feature.

As outlined above, due to diffraction grating a plurality of reflectionfeatures, e.g. for each illumination feature one real feature and aplurality of false features, are generated. The matching is performedwith decreasing brightness of the reflection features starting with thebrightest reflection feature. No other reflection feature can beassigned to the same matched illumination feature. In due of the displayartifacts, the false features which are generated are generally darkerthan the real features. By sorting the reflection features bybrightness, brighter reflection features are preferred for thecorrespondence matching. If a correspondence of an illumination featureis already used, a false feature cannot be assigned to a used, i.e.matched, illumination feature.

The evaluation device is configured for classifying a reflection featurebeing matched with an illumination feature as real feature and forclassifying a reflection feature not being matched with an illuminationfeature as false feature. As used herein, the term “classify” may referto assigning the reflection feature to at least one category. As usedherein, the term “real feature” may refer to a reflection feature ofzero order of diffraction grating. As used herein, the term “falsefeature” may refer to a reflection feature of higher order ofdiffraction grating, i.e. with order Zero order of diffraction gratingare always brighter than false features with a higher order.

The evaluation device is configured for rejecting the false features andfor generating a depth map for the real features by using thelongitudinal coordinate z_(DPR). As used herein, the term “depth” mayrefer to a distance between the object and the optical sensor and may begiven by the longitudinal coordinate. As used herein, the term “depthmap” may refer to spatial distribution of depth. The display device maybe used to generate a 3D map from a scene, e.g. of a face.

Structured light methods commonly use a camera and a projector with afine point grid, e.g. several thousand points. A well-known projectorpattern is used to find the correspondence of point patches on thescene. The distance information is achieved by triangulation if thecorrespondences of the points are solved. If the camera is behind thedisplay, then the diffraction distorts the image spatially. Therefore,it is a challenging task to find point pattern on the distorted image.In comparison to structured light methods, the present inventionproposes using the depth-from-photon-ratio technique for evaluating thebeam profile which are not directly influenced by the diffractiongrating of the display. The distortion does not touch the beam profile.

The depth map can be further refined by using a further depthmeasurement technique such as triangulation and/or depth-from-defocusand/or structured light. The evaluation device may be-configured fordetermining at least one second longitudinal coordinate z_(triang) foreach of the reflection features using triangulation and/ordepth-from-defocus and/or structured light techniques.

The evaluation device may be adapted to determine a displacement of theillumination feature and the reflection feature. The evaluation devicemay be adapted to determine the displacement of the matched illuminationfeature and the selected reflection feature. The evaluation device, e.g.at least one data processing device of the evaluation device, may beconfigured to determine the displacement of the illumination feature andthe reflection feature, in particular by comparing the respective imageposition of the illumination image and the first image. As used herein,the term “displacement” refers to the difference between an imageposition in the illumination image to an image position in the firstimage. The evaluation device may be adapted to determine the secondlongitudinal coordinate of the matched feature using a predeterminedrelationship between the second longitudinal coordinate and thedisplacement. The evaluation device may be adapted to determine thepre-determined relationship by using triangulation methods. In case theposition of the selected reflection feature in the first image and theposition of the matched illumination feature and/or the relativedisplacement of the selected reflection feature and the matchedillumination feature are known, the longitudinal coordinate of thecorresponding object feature may be determined by triangulation. Thus,the evaluation device may be adapted to select, for example subsequentand/or column by column, a reflection feature and to determine for eachpotential position of the illumination feature the correspondingdistance value using triangulation. The displacement and thecorresponding distance value may be stored in at least one storagedevice of the evaluation device. The evaluation device may, as anexample, may comprise at least one data processing device, such as atleast one processor, at least one DSP, at least one FPGA and/or at leastone ASIC. Further, for storing the at least one predetermined ordeterminable relationship between the second longitudinal coordinate zand the displacement, the at least one data storage device may beprovided, such as for providing one or more look-up tables for storingthe predetermined relationship. The evaluation device may be adapted tostore parameters for an intrinsic and/or extrinsic calibration of thecamera and/or the display device. The evaluation device may be adaptedto generate the parameters for an intrinsic and/or extrinsic calibrationof the camera and/or the display device such as by performing a Tsaicamera calibration. The evaluation device may be adapted to computeand/or estimate parameters such as the focal length of the transferdevice, the radial lens distortion coefficient, the coordinates of thecenter of radial lens distortion, scale factors to account for anyuncertainty due to imperfections in hardware timing for scanning anddigitization, rotation angles for the transformation between the worldand camera coordinates, translation components for the transformationbetween the world and camera coordinates, aperture angles, image sensorformat, principal point, skew coefficients, camera center, cameraheading, baseline, rotation or translation parameters between cameraand/or illumination source, apertures, focal distance, or the like.

The evaluation device may be configured for determining a combinedlongitudinal coordinate of the second longitudinal coordinate z_(triang)and the longitudinal coordinate z_(DPR). The combined longitudinalcoordinate may be a mean value of the second longitudinal coordinatez_(triang) and the longitudinal coordinate z_(DPR). The combinedlongitudinal coordinate may be used for determining the depth map.

The display device may comprise a further illumination source. Thefurther illumination source may comprise at least one light emittingdiode (LED). The further illumination source may be configured forgenerating light in the visual spectral range. The optical sensor may beconfigured for determining at least one second image comprising at leastone two dimensional image of the scene. The further illumination sourcemay be configured for providing additional illumination for imaging ofthe second image. For example, the setup of the display device can beextended by an additional flood illumination LED. The furtherillumination source may illuminate the scene, such as a face, with theLED and, in particular, without the illumination pattern, and theoptical sensor may be configured for capturing the two-dimensionalimage. The 2D image may be used for face detection and verificationalgorithm. The distorted image captured by the optical sensor can berepaired, if an impulse response of the display is known. The evaluationdevice may be configured for determining at least one corrected image I₀by deconvoluting the second image I with a grating function g, whereinI=I₀*g. The grating function is also denoted impulse response. Theundistorted image can be restored by a deconvolution approach, e.g.Van-Cittert or Wiener Deconvolution. The display device may beconfigured for determining the grating function g. For example, thedisplay device may be configured for illuminating a black scene with anillumination pattern comprising a small single bright spot. The capturedimage may be the grating function. This procedure may be performed onlyonce such as during calibration. For determining a corrected image evenfor imaging through the display, the display device may be configuredfor capturing the image and use the deconvolution approach with thecaptured impulse response g. The resulting image may be a reconstructedimage with less artifacts of the display and can be used for severalapplications, e.g. face recognition.

The evaluation device may be configured for determining at least onematerial property m of the object by evaluating the beam profile of atleast one of the reflection features, preferably beam profiles of aplurality of reflection features. With respect to details of determiningat least one material property by evaluating the beam profile referenceis made to WO 2020/187719 the content of which is included by reference.

As used herein, the term “material property” refers to at least onearbitrary property of the material configured for characterizing and/oridentification and/or classification of the material. For example, thematerial property may be a property selected from the group consistingof: roughness, penetration depth of light into the material, a propertycharacterizing the material as biological or non-biological material, areflectivity, a specular reflectivity, a diffuse reflectivity, a surfaceproperty, a measure for translucence, a scattering, specifically aback-scattering behavior or the like. The at least one material propertymay be a property selected from the group consisting of: a scatteringcoefficient, a translucency, a transparency, a deviation from aLambertian surface reflection, a speckle, and the like. As used herein,the term “identifying at least one material property” refers to one ormore of determining and assigning the material property to the object.The evaluation device may comprise at least one database comprising alist and/or table, such as a lookup list or a lookup table, ofpredefined and/or predetermined material properties. The list and/ortable of material properties may be determined and/or generated byperforming at least one test measurement using the display deviceaccording to the present invention, for example by performing materialtests using samples having known material properties. The list and/ortable of material properties may be determined and/or generated at themanufacturer site and/or by the user of the display device. The materialproperty may additionally be assigned to a material classifier such asone or more of a material name, a material group such as biological ornon-biological material, translucent or non-translucent materials, metalor nonmetal, skin or non-skin, fur or non-fur, carpet or non-carpet,reflective or non-reflective, specular reflective or non-specularreflective, foam or non-foam, hair or non-hair, roughness groups or thelike. The evaluation device may comprise at least one databasecomprising a list and/or table comprising the material properties andassociated material name and/or material group.

For example, without wishing to be bound by this theory, human skin mayhave a reflection profile, also denoted back scattering profile,comprising parts generated by back reflection of the surface, denoted assurface reflection, and parts generated by very diffuse reflection fromlight penetrating the skin, denoted as diffuse part of the backreflection. With respect to reflection profile of human skin referenceis made to “Lasertechnik in der Medizin: Grundlagen, Systeme,Anwendungen”, “Wirkung von Laserstrahlung auf Gewebe”, 1991, pages 10171 to 266, Jürgen Eichler, Theo Seiler, Springer Verlag, ISBN0939-0979. The surface reflection of the skin may increase with thewavelength increasing towards the near infrared. Further, thepenetration depth may increase with increasing wavelength from visibleto near infrared. The diffuse part of the back reflection may increasewith penetrating depth of the light. These properties may be used todistinguish skin from other materials, by analyzing the back scatteringprofile.

Specifically, the evaluation device may be configured for comparing thebeam profile of the reflection feature, also denoted reflection beamprofile, with at least one predetermined and/or prerecorded and/orpredefined beam profile. The predetermined and/or prerecorded and/orpre-defined beam profile may be stored in a table or a lookup table andmay be determined e.g. empirically, and may, as an example, be stored inat least one data storage device of the display device. For example, thepredetermined and/or prerecorded and/or predefined beam profile may bedetermined during initial start-up of a mobile device comprising thedisplay device. For example, the predetermined and/or prerecorded and/orpredefined beam profile may be stored in at least one data storagedevice of the mobile device, e.g. by software, specifically by the appdownloaded from an app store or the like. The reflection feature may beidentified as to be generated by biological tissue in case thereflection beam profile and the predetermined and/or prerecorded and/orpredefined beam profile are identical. The comparison may compriseoverlaying the reflection beam profile and the predetermined orpredefined beam profile such that their centers of intensity match. Thecomparison may comprise determining a deviation, e.g. a sum of squaredpoint to point distances, between the reflection beam profile and thepre-determined and/or prerecorded and/or predefined beam profile. Theevaluation device may be configured for comparing the determineddeviation with at least one threshold, wherein in case the determineddeviation is below and/or equal the threshold the surface is indicatedas biological tissue and/or the detection of biological tissue isconfirmed. The threshold value may be stored in a table or a lookuptable and may be determined e.g. empirically and may, as an example, bestored in at least one data storage device of the display device.

Additionally or alternatively, for identification if the reflectionfeature was generated by biological tissue, the evaluation device may beconfigured for applying at least one image filter to the image of thearea. As further used herein, the term “image” refers to atwo-dimensional function, f(x,y), wherein brightness and/or color valuesare given for any x,y-position in the image. The position may bediscretized corresponding to the recording pixels. The brightness and/orcolor may be discretized corresponding to a bit-depth of the opticalsensor. As used herein, the term “image filter” refers to at least onemathematical operation applied to the beam profile and/or to the atleast one specific region of the beam profile. Specifically, the imagefilter ϕ maps an image f, or a region of interest in the image, onto areal number, ϕ(f(x,y))=φ, wherein φ denotes a feature, in particular amaterial feature. Images may be subject to noise and the same holds truefor features. Therefore, features may be random variables. The featuresmay be normally distributed. If features are not normally distributed,they may be transformed to be normally distributed such as by aBox-Cox-Transformation.

The evaluation device may be configured for determining at least onematerial feature φ_(2m) by applying at least one material dependentimage filter ϕ2 to the image. As used herein, the term “materialdependent” image filter refers to an image having a material dependentoutput. The output of the material dependent image filter is denotedherein “material feature φ_(2m)” or “material dependent feature φ_(2m)”.The material feature may be or may comprise at least one informationabout the at least one material property of the surface of the areahaving generated the reflection feature.

The material dependent image filter may be at least one filter selectedfrom the group consisting of: a luminance filter; a spot shape filter; asquared norm gradient; a standard deviation; a smoothness filter such asa Gaussian filter or median filter; a grey-level-occurrence-basedcontrast filter; a grey-level-occurrence-based energy filter; agrey-level-occurrence-based homogeneity filter; agrey-level-occurrence-based dissimilarity filter; a Law's energy filter;a threshold area filter; or a linear combination thereof; or a furthermaterial dependent image filter ϕ_(2other) which correlates to one ormore of the luminance filter, the spot shape filter, the squared normgradient, the standard deviation, the smoothness filter, thegrey-level-occurrence-based energy filter, thegrey-level-occurrence-based homogeneity filter, thegrey-level-occurrence-based dissimilarity filter, the Law's energyfilter, or the threshold area filter, or a linear combination thereof by|ρ_(ϕ2other,ϕm)|0.40 with ϕ_(m) being one of the luminance filter, thespot shape filter, the squared norm gradient, the standard deviation,the smoothness filter, the grey-level-occurrence-based energy filter,the grey-level-occurrence-based homogeneity filter, thegrey-level-occurrence-based dissimilarity filter, the Law's energyfilter, or the threshold area filter, or a linear combination thereof.The further material dependent image filter ϕ_(2other) may correlate toone or more of the material dependent image filters Om by|ρ_(ϕ2other,ϕm)|≥0.60, preferably by |ρ_(ϕ2other,ϕm)|≥0.80.

The material dependent image filter may be at least one arbitrary filter(1) that passes a hypothesis testing. As used herein, the term “passes ahypothesis testing” refers to the fact that a Null-hypothesis H₀ isrejected and an alternative hypothesis H₁ is accepted. The hypothesistesting may comprise testing the material dependency of the image filterby applying the image filter to a predefined data set. The data set maycomprise a plurality of beam profile images. As used herein, the term“beam profile image” refers to a sum of N_(B) Gaussian radial basisfunctions,

fk(x,Y)=|Σ_(l=0) ^(NB-1) glk(x,y)|,

g _(lk)(x,y)=a _(lk) e ^(−(α(x-x) _(lk)))² e ⁻(α(y−y _(lk)))²

wherein each of the N_(B) Gaussian radial basis functions is defined bya center (x_(lk), y_(lk)), a prefactor, a_(lk), and an exponentialfactor α=1/∈. The exponential factor is identical for all Gaussianfunctions in all images. The center-positions, x_(lk), y_(lk), areidentical for all images ƒ_(k):(x₀, x₁, . . . ,x_(NB-1)), (y₀,y₁, . . .,y_(NB-1)). Each of the beam profile images in the dataset maycorrespond to a material classifier and a distance. The materialclassifier may be a label such as ‘Material A’, ‘Material B’, etc. Thebeam profile images may be generated by using the above formula forƒ_(k)(x,y) in combination with the following parameter table:

Image Material classifier, Distance Index Material Index z Parameters k= 0 Skin, m = 0 0.4 m (a₀₀, a₁₀, . . . , a_(N) _(B) ⁻¹⁰) k = 1 Skin, m =0 0.6 m (a₀₁, a₁₁, . . . , a_(N) _(B) ⁻¹¹) k = 2 Fabric, m = 1 0.6 m(a₀₂, a₁₂, . . . , a_(N) _(B) ⁻¹²) . . . . . . k = N Material J, m = J −1 (a_(0N), a_(1N), . . . , a_(N) _(B) _(−1N))

The values for x, y, are integers corresponding to pixels with

$\begin{pmatrix}x \\y\end{pmatrix} \in {\left\lbrack {0,1,{\ldots\ 31}} \right\rbrack^{2}.}$

The images may have a pixel size of 32×32. The dataset of beam profileimages may be generated by using the above formula for ƒ_(k) incombination with a parameter set to obtain a continuous description ofƒ_(k). The values for each pixel in the 32×32-image may be obtained byinserting integer values from 0, . . . , 31 for x, y, in ƒ_(k)(x,y). Forexample, for pixel (6,9), the value ƒ_(k) (6,9) may be computed.

Subsequently, for each image ƒ_(k), the feature value φ_(k)corresponding to the filter D may be calculated,ϕ(ƒ_(k)(x,y),z_(k))=(₁)_(k), wherein z_(k) is a distance valuecorresponding to the image ƒ_(k) from the predefined data set. Thisyields a dataset with corresponding generated feature values φ_(k). Thehypothesis testing may use a Null-hypothesis that the filter does notdistinguish between material classifier. The Null-Hypothesis may begiven by H₀: μ₁=μ₂= . . . =μ_(j), wherein μ_(m), is the expectationvalue of each material-group corresponding to the feature values φ_(k).Index m denotes the material group. The hypothesis testing may use asalternative hypothesis that the filter does distinguish between at leasttwo material classifiers. The alternative hypothesis may be given by H₁:∃m, m′:μ_(m)≠μ_(m)′. As used herein, the term “not distinguish betweenmaterial classifiers” refers to that the expectation values of thematerial classifiers are identical. As used herein, the term“distinguishes material classifiers” refers to that at least twoexpectation values of the material classifiers differ. As used herein“distinguishes at least two material classifiers” is used synonymous to“suitable material classifier”. The hypothesis testing may comprise atleast one analysis of variance (ANOVA) on the generated feature values.In particular, the hypothesis testing may comprise determining amean-value of the feature values for each of the J materials, i.e. intotal J mean values,

${{\overset{¯}{\varphi}}_{m} = \frac{\sum\limits_{i}\varphi_{i,m}}{N_{m}}},$

for m ∈[0,1, . . . ,J-1], wherein N_(m) gives the number of featurevalues for each of the J materials in the predefined data set. Thehypothesis testing may comprise determining a mean-value of all Nfeature values

$\overset{\_}{\varphi} = {\frac{\sum_{m}{\sum_{i}\varphi_{i,m}}}{N}.}$

The hypothesis testing may comprise determining a Mean Sum Squareswithin:

mssw=(Σ_(m)Σ_(i)(φ_(i,m)-φ _(m))²)/(N-j).

The hypothesis testing may comprise determining a Mean Sum of Squaresbetween,

mssb=(Σ_(m)(φ _(m)-φ)² N _(m))/(J-1).

The hypothesis testing may comprise performing an F-Test:

${{{{CDF}(x)} = {I_{\frac{d_{1}x}{{d_{1}x} + d_{2}}}\left( {\frac{d_{1}}{2},\frac{d_{2}}{2}} \right)}},{{{where}d_{1}} = {N - J}},{d_{2} = {J - 1}},{{F(x)} = {1 - {{CDF}(x)}}}}{p = {F\left( {{mssb}/{mssw}} \right)}}$

Herein, I_(x) is the regularized incomplete Beta-Function,

${{I_{x}\left( {a,b} \right)} = \frac{B\left( {{x;a},b} \right)}{B\left( {a,b} \right)}},$

with the Euler beta-Function B(a,b)=∫₀ ¹t^(a-1)(1−t)^(b-1)dt and B(x; a,b)=∫₀ ^(x) t^(a-1)(1−t)^(b-1)dt being the incomplete Beta-Function. Theimage filter may pass the hypothesis testing if a p-value, p, is smalleror equal than a pre-defined level of significance. The filter may passthe hypothesis testing if p≤0.075, preferably p≤0.05, more preferablyp≤0.025, and most preferably p≤0.01. For example, in case thepre-defined level of significance is α=0.075, the image filter may passthe hypothesis testing if the p-value is smaller than α=0.075. In thiscase the Null-hypothesis H₀ can be rejected and the alternativehypothesis H₁ can be accepted. The image filter thus distinguishes atleast two material classifiers. Thus, the image filter passes thehypothesis testing.

In the following, image filters are described assuming that thereflection image comprises at least one reflection feature, inparticular a spot image. A spot image ƒ may be given by a function ƒ:

²→

_(≥0), wherein the background of the image ƒ may be already subtracted.However, other reflection features may be possible.

For example, the material dependent image filter may be a luminancefilter. The luminance filter may return a luminance measure of a spot asmaterial feature. The material feature may be determined by

${\varphi_{2m} = {{\Phi\left( {f,z} \right)} = {- {\int{{f(x)}{dx}\frac{z^{2}}{d_{ray} \cdot n}}}}}},$

where f is the spot image. The distance of the spot is denoted by z,where z may be obtained for example by using a depth-from-defocus ordepth-from-photon ratio technique and/or by using a triangulationtechnique. The surface normal of the material is given by n ∈

³ and can be obtained as the normal of the surface spanned by at leastthree measured points. The vector d_(ray) ∈

³ is the direction vector of the light source. Since the position of thespot is known by using a depth-from-defocus or depth-from-photon ratiotechnique and/or by using a triangulation technique wherein the positionof the light source is known as a parameter of the display device,d_(ray), is the difference vector between spot and light sourcepositions.

For example, the material dependent image filter may be a filter havingan output dependent on a spot shape. This material dependent imagefilter may return a value which correlates to the translucence of amaterial as material feature. The translucence of materials influencesthe shape of the spots. The material feature may be given by

${\varphi_{2m} = {{\Phi(f)} = \frac{\int{{H\left( {{f(x)} - {\alpha h}} \right)}{dx}}}{\int{{H\left( {{f(x)} - {\beta h}} \right)}{dx}}}}},$

wherein 0<α,β<1 are weights for the spot height h, and H denotes theHeavyside function, i.e. H(x)=1: x≥0, H(x)=0: x<0. The spot height h maybe determined by

h=∫ _(Br)ƒ(x)dx,

where B_(r) is an inner circle of a spot with radius r.

For example, the material dependent image filter may be a squared normgradient. This material dependent image filter may return a value whichcorrelates to a measure of soft and hard transitions and/or roughness ofa spot as material feature. The material feature may be defined by

φ_(2m)=Φ(ƒ)=∫∥∇ƒ(x)∥² dx.

For example, the material dependent image filter may be a standarddeviation. The standard deviation of the spot may be determined by

φ_(2m)=Φ(ƒ)=∫(ƒ(x)−μ)² dx,

Wherein μ is the mean value given by μ=∫(ƒ(x))dx.

For example, the material dependent image filter may be a smoothnessfilter such as a Gaussian filter or median filter. In one embodiment ofthe smoothness filter, this image filter may refer to the observationthat volume scattering exhibits less speckle contrast compared todiffuse scattering materials. This image filter may quantify thesmoothness of the spot corresponding to speckle contrast as materialfeature. The material feature may be determined by

${\varphi_{2m} = {{\Phi\left( {f,z} \right)} = {\frac{\int{{❘{{{\mathcal{F}(f)}(x)} - {f(x)}}❘}{dx}}}{\int{{f(x)}{dx}}} \cdot \frac{1}{z}}}},$

wherein

is a smoothness function, for example a median filter or Gaussianfilter. This image filter may comprise dividing by the distance z, asdescribed in the formula above. The distance z may be determined forexample using a depth-from-defocus or depth-from-photon ratio techniqueand/or by using a triangulation technique. This may allow the filter tobe insensitive to distance. In one embodiment of the smoothness filter,the smoothness filter may be based on the standard deviation of anextracted speckle noise pattern. A speckle noise pattern N can bedescribed in an empirical way by

ƒ(x)=ƒ₀(x)·(N(X)+1),

where ƒ₀ is an image of a despeckled spot. N(X) is the noise term thatmodels the speckle pattern. The computation of a despeckled image may bedifficult. Thus, the despeckled image may be approximated with asmoothed version of f, i.e. ƒ₀≈

(ƒ), wherein

is a smoothness operator like a Gaussian filter or median filter. Thus,an approximation of the speckle pattern may be given by

${N(X)} = {\frac{f(x)}{\mathcal{F}\left( {f(x)} \right)} - 1.}$

The material feature of this filter may be determined by

${\varphi_{2m} = {{\Phi(f)} = \sqrt{{Var}\left( {\frac{f}{\mathcal{F}(f)} - 1} \right)}}},$

Wherein Var denotes the variance function.

For example, the image filter may be a grey-level-occurrence-basedcontrast filter. This material filter may be based on the grey leveloccurrence matrix M_(ƒ,p) (g₁,g₂)=[p_(g1,g2)], whereas p_(g1,g2) is theoccurrence rate of the grey combination (g₁,g₂)=[f(x₁,y₁),f(x₂,y₂)], andthe relation p defines the distance between (x₁,y₁) and (x₂,y₂), whichis p(x,y)=(x+a,y+b) with a and b selected from 0,1.

The material feature of the grey-level-occurrence-based contrast filtermay be given by

$\varphi_{2m} = {{\Phi(f)} = {\sum\limits_{i,{j = 0}}^{N - 1}{{p_{ij}\left( {i - j} \right)}^{2}.}}}$

For example, the image filter may be a grey-level-occurrence-basedenergy filter. This material filter is based on the grey leveloccurrence matrix defined above.

The material feature of the grey-level-occurrence-based energy filtermay be given by

$\varphi_{2m} = {{\Phi(f)} = {\sum\limits_{i,{j = 0}}^{N - 1}{\left( p_{ij} \right)^{2}.}}}$

For example, the image filter may be a grey-level-occurrence-basedhomogeneity filter. This material filter is based on the grey leveloccurrence matrix defined above.

The material feature of the grey-level-occurrence-based homogeneityfilter may be given by

$\varphi_{2m} = {{\Phi(f)} = {\sum\limits_{i,{j = 0}}^{N - 1}{\frac{p_{ij}}{1 + {❘{i - j}❘}}.}}}$

For example, the image filter may be a grey-level-occurrence-baseddissimilarity filter. This material filter is based on the grey leveloccurrence matrix defined above.

The material feature of the grey-level-occurrence-based dissimilarityfilter may be given by

$\varphi_{2m} = {{\Phi(f)} = {- {\sum\limits_{i,{j = 0}}^{N - 1}{\sqrt{p_{ij}{\log\left( p_{ij} \right)}}.}}}}$

For example, the image filter may be a Law's energy filter. Thismaterial filter may be based on the laws vector L₅=[1,4,6,4,1] andE₅=[−1,−2,0,−2,−1] and the matrices L₅(E₅)^(T) and E₅(L₅)^(T).

The image ƒ_(k) is convoluted with these matrices:

${{f_{k,{L5E5}}^{*}\left( {x,y} \right)} = {\sum\limits_{i - 2}^{2}{\sum\limits_{j - 2}^{2}{{f_{k}\left( {{x + i},{y + j}} \right)}{L_{5}\left( E_{5} \right)}^{T}{and}}}}}{{{f_{k,{E5L5}}^{*}\left( {x,y} \right)} = {{\sum_{i - 2}^{2}{\sum_{j - 2}^{2}{{f_{k}\left( {{x + i},{y + j}} \right)}{{E_{5}\left( L_{5} \right)}^{T}.E}}}} = {\int{\frac{f_{k,{L5E5}}^{*}\left( {x,y} \right)}{\max\left( {f_{k,{L5E5}}^{*}\left( {x,y} \right)} \right)}{dxdy}}}}},{F = {\int{\frac{f_{k,{E5L5}}^{*}\left( {x,y} \right)}{\max\left( {f_{k,{E5L5}}^{*}\left( {x,y} \right)} \right)}{dxdy}}}},}$

Whereas the material feature of Law's energy filter may be determined by

φ_(2m)=Φ(ƒ)=E/F.

For example, the material dependent image filter may be a threshold areafilter. This material feature may relate two areas in the image plane. Afirst area Ω1, may be an area wherein the function ƒ is larger than αtimes the maximum of f. A second area Ω2, may be an area wherein thefunction ƒ is smaller than α times the maximum of f, but larger than athreshold value £ times the maximum of f. Preferably a may be 0.5 and εmay be 0.05. Due to speckles or noise, the areas may not simplycorrespond to an inner and an outer circle around the spot center. As anexample, Ω1 may comprise speckles or unconnected areas in the outercircle. The material feature may be determined by

${\varphi_{2m} = {{\Phi(f)} = \frac{\int_{\Omega 1}1}{\int_{\Omega 2}1}}},$

wherein Ω1={x|f(x)>α·max(f(x))} and Ω2={x|ε·max(f(x))<f(x)<α·max(f(x))}.

The evaluation device may be configured for using at least onepredetermined relationship between the material feature φ_(2m) and thematerial property of the surface having generated the reflection featurefor determining the material property of the surface having generatedthe reflection feature. The predetermined relationship may be one ormore of an empirical relationship, a semi-empiric relationship and ananalytically derived relationship. The evaluation device may comprise atleast one data storage device for storing the predeterminedrelationship, such as a lookup list or a lookup table.

The evaluation device is configured for identifying a reflection featureas to be generated by illuminating biological tissue in case itscorresponding material property fulfills the at least one pre-determinedor predefined criterion. The reflection feature may be identified as tobe generated by biological tissue in case the material propertyindicates “biological tissue”. The reflection feature may be identifiedas to be generated by biological tissue in case the material property isbelow or equal at least one threshold or range, wherein in case thedetermined deviation is below and/or equal the threshold the reflectionfeature is identified as to be generated by biological tissue and/or thedetection of biological tissue is confirmed. At least one thresholdvalue and/or range may be stored in a table or a lookup table and may bedetermined e.g. empirically and may, as an example, be stored in atleast one data storage device of the display device. The evaluationdevice is configured for identifying the reflection feature as to bebackground otherwise. Thus, the evaluation device may be configured forassigning each projected spot with a depth information and a materialproperty, e.g. skin yes or no.

The material property may be determined by evaluating φ_(2m)subsequently after determining of the longitudinal coordinate z suchthat the information about the longitudinal coordinate z can beconsidered for evaluating of φ_(2m).

In a further aspect, the present invention discloses a method for depthmeasurement through a translucent display, wherein a display deviceaccording to the present invention is used. The method comprises thefollowing steps:

-   -   a) projecting at least one illumination pattern comprising a        plurality of illumination features on at least one scene by        using at least one illumination source, wherein the illumination        source is placed in direction of propagation of the illumination        pattern in front of the display;    -   b) determining at least one first image comprising a plurality        of reflection features generated by the scene in response to        illumination by the illumination features by using at least one        optical sensor, wherein the optical sensor has at least one        light sensitive area, wherein the optical sensor is placed in        direction of propagation of the illumination pattern in front of        the display, wherein each of the reflection features comprises        at least one beam profile;    -   c) Evaluating the first image by using at least one evaluation        device, wherein the evaluation comprises the following substeps:        -   C1) identifying the reflection features of the first image            and sorting the identified reflection features with respect            to brightness;        -   C2) determining at least one longitudinal coordinate z_(DPR)            for each of the reflection features by analysis of their            beam profiles;        -   C3) unambiguously matching of reflection features with            corresponding illumination features by using the            longitudinal coordinate z_(DPR), wherein the matching is            performed with decreasing brightness of the reflection            features starting with the brightest reflection feature;        -   C4) classifying a reflection feature being matched with an            illumination feature as real feature and for classifying a            reflection feature not being matched with an illumination            feature as false feature;        -   C5) rejecting the false features and for generating the            depth map for the real features by using the longitudinal            coordinate z_(DPR).

The method steps may be performed in the given order or may be performedin a different order. Further, one or more additional method steps maybe present which are not listed. Further, one, more than one or even allof the method steps may be performed repeatedly. For details, optionsand definitions, reference may be made to the display device asdiscussed above. Thus, specifically, as outlined above, the method maycomprise using the display device according to the present invention,such as according to one or more of the embodiments given above or givenin further detail below.

The at least one evaluation device may be configured for performing atleast one computer program, such as at least one computer programconfigured for performing or supporting one or more or even all of themethod steps of the method according to the present invention. As anexample, one or more algorithms may be implemented which may determinethe position of the object.

In a further aspect of the present invention, use of the display deviceaccording to the present invention, such as according to one or more ofthe embodiments given above or given in further detail below, isproposed, for a purpose of use, selected from the group consisting of: aposition measurement in traffic technology; an entertainmentapplication; a security application; a surveillance application; asafety application; a human-machine interface application; a trackingapplication; a photography application; an imaging application or cameraapplication; a mapping application for generating maps of at least onespace; a homing or tracking beacon detector for vehicles; an outdoorapplication; a mobile application; a communication application; amachine vision application; a robotics application; a quality controlapplication; a manufacturing application.

With respect to further uses of the display device and devices of thepresent invention reference is made to WO 2018/091649 A1, WO 2018/091638A1 and WO 2018/091640 A1, the content of which is included by reference.

Overall, in the context of the present invention, the followingembodiments are regarded as preferred:

Embodiment 1: A display device comprising

-   -   at least one illumination source configured for projecting at        least one illumination pattern comprising a plurality of        illumination features on at least one scene;    -   at least one optical sensor having at least one light sensitive        area, wherein the optical sensor is configured for determining        at least one first image comprising a plurality of reflection        features generated by the scene in response to illumination by        the illumination features;    -   at least one translucent display configured for displaying        information, wherein the illumination source and the optical        sensor are placed in direction of propagation of the        illumination pattern in front of the display;    -   at least one evaluation device, wherein the evaluation device is        configured for evaluating the first image, wherein the        evaluation of the first image comprises identifying the        reflection features of the first image and sorting the        identified reflection features with respect to brightness,        wherein each of the reflection features comprises at least one        beam profile, wherein the evaluation device is configured for        determining at least one longitudinal coordinate z_(DPR) for        each of the reflection features by analysis of their beam        profiles,

wherein the evaluation device is configured for unambiguously matchingof reflection features with corresponding illumination features by usingthe longitudinal coordinate z_(DPR), wherein the matching is performedwith decreasing brightness of the reflection features starting with thebrightest reflection feature, wherein the evaluation device isconfigured for classifying a reflection feature being matched with anillumination feature as real feature and for classifying a reflectionfeature not being matched with an illumination feature as false feature,wherein the evaluation device is configured for rejecting the falsefeatures and for generating a depth map for the real features by usingthe longitudinal coordinate z_(DPR).

Embodiment 2: The display device according to the preceding embodiment,wherein the evaluation device is configured for determining at least onesecond longitudinal coordinate z_(triang) for each of the reflectionfeatures using triangulation and/or depth-from-defocus and/or structuredlight techniques.

Embodiment 3: The display device according to the preceding embodiment,wherein the evaluation device is configured for determining a combinedlongitudinal coordinate of the second longitudinal coordinate z_(triang)and the longitudinal coordinate z_(DPR), wherein the combinedlongitudinal coordinate is a mean value of the second longitudinalcoordinate z_(triang) and the longitudinal coordinate z_(DPR), whereinthe combined longitudinal coordinate is used for determining the depthmap.

Embodiment 4: The display device according to any one of the precedingembodiments, wherein the illumination source comprises at least onelaser projector, wherein the laser projector comprises at least onelaser source and at least one diffractive optical element (DOE).

Embodiment 5: The display device according to any one of the precedingembodiments, wherein the illumination source is configured forgenerating at least one light beam having a beam path passing from theillumination source through the display to the scene, wherein thedisplay is configured for functioning as grating such that the lightbeam experiences diffraction by the display which results in the pointpattern.

Embodiment 6: The display device according to the preceding embodiment,wherein a wiring of the display is configured for forming gaps and/orslits and ridges of the grating.

Embodiment 7: The display device according to any one of the precedingembodiments, wherein the illumination pattern comprises a periodic pointpattern.

Embodiment 8: The display device according to any one of the precedingembodiments, wherein the illumination pattern has a low point density,wherein the illumination pattern has 2500 points per field of view.

Embodiment 9: The display device according to any one of the precedingembodiments, wherein the evaluation device is configured for determiningthe beam profile information for each of the reflection features byusing depth-from-photon-ratio technique.

Embodiment 10: The display device according to any one of the precedingembodiments, wherein the optical sensor comprises at least one CMOSsensor.

Embodiment 11: The display device according to any one of the precedingembodiments, wherein the display device comprises a further illuminationsource, wherein the further illumination source comprises at least onelight emitting diode (LED).

Embodiment 12: The display device according to the preceding embodiment,wherein the further illumination source is configured for generatinglight in the visual spectral range.

Embodiment 13: The display device according to any one of the twopreceding embodiments, wherein the optical sensor is configured fordetermining at least one second image comprising at least one twodimensional image of the scene, wherein the further illumination sourceis configured for providing additional illumination for imaging of thesecond image.

Embodiment 14: The display device according to the preceding embodiment,wherein the evaluation device is configured for determining at least onecorrected image I₀ by deconvoluting the second image I with a gratingfunction g, wherein I=I₀*g.

Embodiment 15: Method for depth measurement through a translucentdisplay, wherein at least one display device according to any one of thepreceding embodiments is used, wherein the method comprises thefollowing steps:

-   -   a) projecting at least one illumination pattern comprising a        plurality of illumination features on at least one scene by        using at least one illumination source, wherein the illumination        source is placed in direction of propagation of the illumination        pattern in front of the display;    -   b) determining at least one first image comprising a plurality        of reflection features generated by the scene in response to        illumination by the illumination features by using at least one        optical sensor, wherein the optical sensor has at least one        light sensitive area, wherein the optical sensor is placed in        direction of propagation of the illumination pattern in front of        the display, wherein each of the reflection features comprises        at least one beam profile;    -   c) Evaluating the first image by using at least one evaluation        device, wherein the evaluation comprises the following substeps:        -   C1) identifying the reflection features of the first image            and sorting the identified reflection features with respect            to brightness;        -   C2) determining at least one longitudinal coordinate z_(DPR)            for each of the reflection features by analysis of their            beam profiles;        -   C3) unambiguously matching of reflection features with            corresponding illumination features by using the            longitudinal coordinate z_(DPR), wherein the matching is            performed with decreasing brightness of the reflection            features starting with the brightest reflection feature;        -   C4) classifying a reflection feature being matched with an            illumination feature as real feature and for classifying a            reflection feature not being matched with an illumination            feature as false feature;        -   C5) rejecting the false features and for generating the            depth map for the real features by using the longitudinal            coordinate z_(DPR).

Embodiment 16: A use of the display device according to any one of thepreceding embodiments relating to a display device, for a purpose ofuse, selected from the group consisting of: a position measurement intraffic technology; an entertainment application; a securityapplication; a surveillance application; a safety application; ahuman-machine interface application; a tracking application; aphotography application; an imaging application or camera application; amapping application for generating maps of at least one space; a homingor tracking beacon detector for vehicles; an outdoor application; amobile application; a communication application; a machine visionapplication; a robotics application; a quality control application; amanufacturing application.

BRIEF DESCRIPTION OF THE FIGURES

Further optional details and features of the invention are evident fromthe description of preferred exemplary embodiments which follows inconjunction with the dependent claims. In this context, the particularfeatures may be implemented in an isolated fashion or in combinationwith other features. The invention is not restricted to the exemplaryembodiments. The exemplary embodiments are shown schematically in thefigures. Identical reference numerals in the individual figures refer toidentical elements or elements with identical function, or elementswhich correspond to one another with regard to their functions.

Specifically, in the figures:

FIGS. 1A and 1B show embodiments of a display device according to thepresent invention;

FIGS. 2A to 2B show embodiments of first images determined with at leastone optical sensor of the display device;

FIGS. 3A to 3C show further embodiments of first images determined withat least one optical sensor of the display device;

FIG. 4 shows determining of a corrected 2D image using the displaydevice; and

FIGS. 5A to 5C show a distorted 2D-image captured with a display, a2D-image captured without the display and a corrected 2D-image.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1A shows in a highly schematic fashion an embodiment of a displaydevice 110 according to the present invention. The display device 110comprises at least one translucent display 112 configured for displayinginformation. The display 112 may be an arbitrary shaped deviceconfigured for displaying an item of information such as at least oneimage, at least one diagram, at least one histogram, at least one text,at least one sign. The display 112 may be at least one monitor or atleast one screen. The display 112 may have an arbitrary shape,preferably a rectangular shape. For example, the display device 110 maybe at least one device selected from the group consisting of: atelevision device, smart phones, game consoles, personal computers,laptops, tablets, at least one virtual reality device, or combinationsthereof.

The display device 110 comprises at least one illumination source 114configured for projecting at least one illumination pattern comprising aplurality of illumination features on at least one scene. The scene maybe an object or spatial region, such as a face. The scene may comprisethe at least one object and a surrounding environment.

The illumination source 114 may be adapted to directly or indirectlyilluminating the scene, wherein the illumination pattern is reflected orscattered by surfaces of the scene and, thereby, is at least partiallydirected towards the optical sensor. The illumination source 114 may beadapted to illuminate the scene, for example, by directing a light beamtowards the scene, which reflects the light beam. The illuminationsource 114 may be configured for generating an illuminating light beamfor illuminating the scene.

The illumination source 114 may comprise at least one light source. Theillumination source 114 may comprise a plurality of light sources. Theillumination source 114 may comprise an artificial illumination source,in particular at least one laser source and/or at least one incandescentlamp and/or at least one semiconductor light source, for example, atleast one light-emitting diode, in particular an organic and/orinorganic light-emitting diode. As an example, the light emitted by theillumination source may have a wavelength of 300 to 1100 nm, especially500 to 1100 nm. Additionally or alternatively, light in the infraredspectral range may be used, such as in the range of 780 nm to 3.0 μm.Specifically, the light in the part of the near infrared region wheresilicon photodiodes are applicable specifically in the range of 700 nmto 1100 nm may be used. The illumination source 114 may be configuredfor generating the at least one illumination pattern in the infraredregion. Using light in the near infrared region allows that light is notor only weakly detected by human eyes and is still detectable by siliconsensors, in particular standard silicon sensors. The illumination source114 may be configured for emitting light at a single wavelength.Specifically, the wavelength may be in the near infrared region. Inother embodiments, the illumination may be adapted to emit light with aplurality of wavelengths allowing additional measurements in otherwavelengths channels

The illumination source 114 may be or may comprise at least one multiplebeam light source. For example, the illumination source 114 may compriseat least one laser source and one or more diffractive optical elements(DOEs). Specifically, the illumination source may comprise at least onelaser and/or laser source. Various types of lasers may be employed, suchas semiconductor lasers, double heterostructure lasers, external cavitylasers, separate confinement heterostructure lasers, quantum cascadelasers, distributed bragg reflector lasers, polariton lasers, hybridsilicon lasers, extended cavity diode lasers, quantum dot lasers, volumeBragg grating lasers, Indium Arsenide lasers, transistor lasers, diodepumped lasers, distributed feedback lasers, quantum well lasers,interband cascade lasers, Gallium Arsenide lasers, semiconductor ringlaser, extended cavity diode lasers, or vertical cavity surface-emittinglasers. Additionally or alternatively, non-laser light sources may beused, such as LEDs and/or light bulbs. The illumination source maycomprise one or more diffractive optical elements (DOEs) adapted togenerate the illumination pattern. For example, the illumination source114 may be adapted to generate and/or to project a cloud of points, forexample the illumination source may comprise one or more of at least onedigital light processing projector, at least one LCoS projector, atleast one spatial light modulator; at least one diffractive opticalelement; at least one array of light emitting diodes; at least one arrayof laser light sources. On account of their generally defined beamprofiles and other properties of handleability, the use of at least onelaser source as the illumination source 114 is particularly preferred.The illumination source 114 may be integrated into a housing 116 of thedisplay device 110.

Further, the illumination source 114 may be configured for emittingmodulated or non-modulated light. In case a plurality of illuminationsources 114 is used, the different illumination sources may havedifferent modulation frequencies which, as outlined in further detailbelow, later on may be used for distinguishing the light beams.

The illumination pattern may be at least one arbitrary patterncomprising at least one illumination feature adapted to illuminate atleast one part of the scene. The illumination pattern may comprise asingle illumination feature. The illumination pattern may comprise aplurality of illumination features. The illumination pattern may beselected from the group consisting of: at least one point pattern; atleast one line pattern; at least one stripe pattern; at least onecheckerboard pattern; at least one pattern comprising an arrangement ofperiodic or non periodic features. The illumination pattern may compriseregular and/or constant and/or periodic pattern such as a triangularpattern, a rectangular pattern, a hexagonal pattern or a patterncomprising further convex tilings. The illumination pattern may exhibitthe at least one illumination feature selected from the group consistingof: at least one point; at least one line; at least two lines such asparallel or crossing lines; at least one point and one line; at leastone arrangement of periodic or non-periodic feature; at least onearbitrary shaped featured. The illumination pattern may comprise atleast one pattern selected from the group consisting of: at least onepoint pattern, in particular a pseudo-random point pattern; a randompoint pattern or a quasi random pattern; at least one Sobol pattern; atleast one quasiperiodic pattern; at least one pattern comprising atleast one pre-known feature at least one regular pattern; at least onetriangular pattern; at least one hexagonal pattern; at least onerectangular pattern at least one pattern comprising convex uniformtilings; at least one line pattern comprising at least one line; atleast one line pattern comprising at least two lines such as parallel orcrossing lines. For example, the illumination source may be adapted togenerate and/or to project a cloud of points. The illumination source114 may comprise the at least one light projector adapted to generate acloud of points such that the illumination pattern may comprise aplurality of point pattern. The illumination source 114 may comprise atleast one mask adapted to generate the illumination pattern from atleast one light beam generated by the illumination source 114.

A distance between two features of the illumination pattern and/or anarea of the at least one illumination feature may depend on the circleof confusion in the image. As outlined above, the illumination sourcemay comprise the at least one light source configured for generating theat least one illumination pattern. Specifically, the illumination source114 comprises at least one laser source and/or at least one laser diodewhich is designated for generating laser radiation. The illuminationsource 114 may comprise the at least one diffractive optical element(DOE). The display device 110 may comprise at least one point projector,such as the at least one laser source and the DOE, adapted to project atleast one periodic point pattern.

For example, the projected illumination pattern may be a periodic pointpattern. The projected illumination pattern may have a low pointdensity. For example, the illumination pattern may comprise at least oneperiodic point pattern having a low point density, wherein theillumination pattern has 2500 points per field of view. In comparisonwith structured light having typically a point density of 10 k-30 k in afield of view of 55×38° the illumination pattern according to thepresent invention may be less dense. This may allow more power per pointsuch that the proposed technique is less dependent on ambient lightcompared to structured light.

The display device 110 comprises at least one optical sensor 118 havingat least one light sensitive area 120. The optical sensor 118 isconfigured for determining at least one first image 122, shown e.g. inFIGS. 2A to 2C and 3A to 3C, comprising a plurality of reflectionfeatures generated by the scene in response to illumination by theillumination features. The display device 110 may comprise a singlecamera comprising the optical sensor 118. The display device 110 maycomprise a plurality of cameras each comprising an optical sensor 118 ora plurality of optical sensors 118.

The optical sensor 118 specifically may be or may comprise at least onephotodetector, preferably inorganic photodetectors, more preferablyinorganic semiconductor photodetectors, most preferably siliconphotodetectors. Specifically, the optical sensor 118 may be sensitive inthe infrared spectral range. All pixels of the matrix or at least agroup of the optical sensors of the matrix specifically may beidentical. Groups of identical pixels of the matrix specifically may beprovided for different spectral ranges, or all pixels may be identicalin terms of spectral sensitivity. Further, the pixels may be identicalin size and/or with regard to their electronic or optoelectronicproperties. Specifically, the optical sensor 118 may be or may compriseat least one inorganic photodiode which are sensitive in the infraredspectral range, preferably in the range of 700 nm to 3.0 micrometers.Specifically, the optical sensor 118 may be sensitive in the part of thenear infrared region where silicon photodiodes are applicablespecifically in the range of 700 nm to 1100 nm. Infrared optical sensorswhich may be used for optical sensors may be commercially availableinfrared optical sensors, such as infrared optical sensors commerciallyavailable under the brand name Hertzstueck™ from trinamiX™ GmbH, D-67056Ludwigshafen am Rhein, Germany. Thus, as an example, the optical sensor118 may comprise at least one optical sensor of an intrinsicphotovoltaic type, more preferably at least one semiconductor photodiodeselected from the group consisting of: a Ge photodiode, an InGaAsphotodiode, an extended InGaAs photodiode, an InAs photodiode, an InSbphotodiode, a HgCdTe photodiode. Additionally or alternatively, theoptical sensor 118 may comprise at least one optical sensor of anextrinsic photovoltaic type, more preferably at least one semiconductorphotodiode selected from the group consisting of: a Ge:Au photodiode, aGe:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Gaphotodiode, a Si:As photodiode. Additionally or alternatively, theoptical sensor 118 may comprise at least one photoconductive sensor suchas a PbS or PbSe sensor, a bolometer, preferably a bolometer selectedfrom the group consisting of a VO bolometer and an amorphous Sibolometer.

The optical sensor 118 may be sensitive in one or more of theultraviolet, the visible or the infrared spectral range. Specifically,the optical sensor may be sensitive in the visible spectral range from500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to700 nm. Specifically, the optical sensor 118 may be sensitive in thenear infrared region. Specifically, the optical sensor 118 may besensitive in the part of the near infrared region where siliconphotodiodes are applicable specifically in the range of 700 nm to 1000nm. The optical sensor 118, specifically, may be sensitive in theinfrared spectral range, specifically in the range of 780 nm to 3.0micrometers. For example, the optical sensor each, independently, may beor may comprise at least one element selected from the group consistingof a photodiode, a photocell, a photoconductor, a phototransistor or anycombination thereof. For example, the optical sensor 118 may be or maycomprise at least one element selected from the group consisting of aCCD sensor element, a CMOS sensor element, a photodiode, a photocell, aphotoconductor, a phototransistor or any combination thereof. Any othertype of photosensitive element may be used. The photosensitive elementgenerally may fully or partially be made of inorganic materials and/ormay fully or partially be made of organic materials. Most commonly, oneor more photodiodes may be used, such as commercially availablephotodiodes, e.g. inorganic semiconductor photodiodes.

The optical sensor 118 may comprise at least one sensor elementcomprising a matrix of pixels. Thus, as an example, the optical sensor118 may be part of or constitute a pixelated optical device. Forexample, the optical sensor 118 may be and/or may comprise at least oneCCD and/or CMOS device. As an example, the optical sensor 118 may bepart of or constitute at least one CCD and/or CMOS device having amatrix of pixels, each pixel forming a light-sensitive area. The sensorelement may be formed as a unitary, single device or as a combination ofseveral devices. The matrix specifically may be or may comprise arectangular matrix having one or more rows and one or more columns. Therows and columns specifically may be arranged in a rectangular fashion.However, other arrangements are feasible, such as nonrectangulararrangements. As an example, circular arrangements are also feasible,wherein the elements are arranged in concentric circles or ellipsesabout a center point. For example, the matrix may be a single row ofpixels. Other arrangements are feasible.

The pixels of the matrix specifically may be equal in one or more ofsize, sensitivity and other optical, electrical and mechanicalproperties. The light-sensitive areas 120 of all optical sensors 118 ofthe matrix specifically may be located in a common plane, the commonplane preferably facing the scene, such that a light beam propagatingfrom the object to the display device 110 may generate a light spot onthe common plane. The light-sensitive area 120 may specifically belocated on a surface of the respective optical sensor 118. Otherembodiments, however, are feasible. The optical sensor 118 may comprisefor example, at least one CCD and/or CMOS device. As an example, theoptical sensor 118 may be part of or constitute a pixelated opticaldevice. As an example, the optical sensor 118 may be part of orconstitute at least one CCD and/or CMOS device having a matrix ofpixels, each pixel forming a light-sensitive area 120.

The display device 110 comprises the at least one translucent display112 configured for displaying information. The illumination source 114and the optical sensor 118 are placed in direction of propagation of theillumination pattern in front of the display 112. The illuminationsource 114 and the optical sensor 118 may be arranged in a fixedposition with respect to each other. For example, the setup of thedisplay device 110 may comprise a camera, comprising the optical sensor118 and a lens system, and a laser projector as illumination source 114.The laser projector and the camera may be fixed, in a direction ofpropagation of light reflected by the scene, behind the translucentdisplay. The laser projector may generate a dot pattern and shinesthrough the display 112. The camera may look through the display. Thearrangement of the illumination source 114 and optical sensor 118 in adirection of propagation of light reflected by the scene, behind thetranslucent display, however, may result in that diffraction grating ofthe display 112 generates multiple laser points on the scene and also inthe first image. Thereby these multiple spots on the first image may notinclude any useful distance information. The display device 110comprises at least on evaluation device 124. The evaluation device 124may be configured for finding and evaluating the reflection features ofthe zero order of diffraction grating, i.e. real features, and mayneglect the reflection features of the higher orders, i.e. falsefeatures.

The evaluation device 124 is configured for evaluating the first image.The evaluation device 124 may comprise at least one data processingdevice and, more preferably, by using at least one processor and/or atleast one application-specific integrated circuit. Thus, as an example,the at least one evaluation device 124 may comprise at least one dataprocessing device having a software code stored thereon comprising anumber of computer commands. The evaluation device 124 may provide oneor more hardware elements for performing one or more of the namedoperations and/or may provide one or more processors with softwarerunning thereon for performing one or more of the named operations.Operations, including evaluating the images. Specifically, thedetermining the beam profile and indication of the surface, may beperformed by the at least one evaluation device. Thus, as an example,one or more instructions may be implemented in software and/or hardware.Thus, as an example, the evaluation device 124 may comprise one or moreprogrammable devices such as one or more computers, application-specificintegrated circuits (ASICs), Digital Signal Processors (DSPs), or FieldProgrammable Gate Arrays (FPGAs) which are configured to perform theabove-mentioned evaluation. Additionally or alternatively, however, theevaluation device may also fully or partially be embodied by hardware.

The evaluation of the first image comprises identifying the reflectionfeatures of the first image. The evaluation device 124 may be configuredfor performing at least one image analysis and/or image processing inorder to identify the reflection features. The image analysis and/orimage processing may use at least one feature detection algorithm. Theimage analysis and/or image processing may comprise one or more of thefollowing: a filtering; a selection of at least one region of interest;a formation of a difference image between an image created by the sensorsignals and at least one offset; an inversion of sensor signals byinverting an image created by the sensor signals; a formation of adifference image between an image created by the sensor signals atdifferent times; a background correction; a decomposition into colorchannels; a decomposition into hue; saturation; and brightness channels;a frequency decomposition; a singular value decomposition; applying ablob detector; applying a corner detector; applying a Determinant ofHessian filter; applying a principle curvature-based region detector;applying a maximally stable extremal regions detector; applying ageneralized Hough-transformation; applying a ridge detector; applying anaffine invariant feature detector; applying an affine-adapted interestpoint operator; applying a Harris affine region detector; applying aHessian affine region detector; applying a scale-invariant featuretransform; applying a scale-space extrema detector; applying a localfeature detector; applying speeded up robust features algorithm;applying a gradient location and orientation histogram algorithm;applying a histogram of oriented gradients descriptor; applying aDeriche edge detector; applying a differential edge detector; applying aspatio-temporal interest point detector; applying a Moravec cornerdetector; applying a Canny edge detector; applying a Laplacian ofGaussian filter; applying a Difference of Gaussian filter; applying aSobel operator; applying a Laplace operator; applying a Scharr operator;applying a Prewitt operator; applying a Roberts operator; applying aKirsch operator; applying a high-pass filter; applying a low-passfilter; applying a Fourier transformation; applying aRadon-transformation; applying a Hough-transformation; applying awavelet-transformation; a thresholding; creating a binary image. Theregion of interest may be determined manually by a user or may bedetermined automatically, such as by recognizing a feature within theimage generated by the optical sensor.

For example, the illumination source 114 may be configured forgenerating and/or projecting a cloud of points such that a plurality ofilluminated regions is generated on the optical sensor 118, for examplethe CMOS detector. Additionally, disturbances may be present on theoptical sensor 118 such as disturbances due to speckles and/orextraneous light and/or multiple reflections. The evaluation device 124may be adapted to determine at least one region of interest, for exampleone or more pixels illuminated by the light beam which are used fordetermination of the longitudinal coordinate of the object. For example,the evaluation device 124 may be adapted to perform a filtering method,for example, a blob-analysis and/or an edge filter and/or objectrecognition method.

The evaluation device 124 may be configured for performing at least oneimage correction. The image correction may comprise at least onebackground subtraction. The evaluation device 124 may be adapted toremove influences from background light from the beam profile, forexample, by an imaging without further illumination.

Each of the reflection features comprises at least one beam profile. Thebeam profile may be selected from the group consisting of a trapezoidbeam profile; a triangle beam profile; a conical beam profile and alinear combination of Gaussian beam profiles. The evaluation device isconfigured for determining beam profile information for each of thereflection features by analysis of their beam profiles.

The evaluation device 124 is configured for determining at least onelongitudinal coordinate z_(DPR) for each of the reflection features byanalysis of their beam profiles. For example, the analysis of the beamprofile may comprise at least one of a histogram analysis step, acalculation of a difference measure, application of a neural network,application of a machine learning algorithm. The evaluation device 124may be configured for symmetrizing and/or for normalizing and/or forfiltering the beam profile, in particular to remove noise or asymmetriesfrom recording under larger angles, recording edges or the like. Theevaluation device 124 may filter the beam profile by removing highspatial frequencies such as by spatial frequency analysis and/or medianfiltering or the like. Summarization may be performed by center ofintensity of the light spot and averaging all intensities at the samedistance to the center. The evaluation device 124 may be configured fornormalizing the beam profile to a maximum intensity, in particular toaccount for intensity differences due to the recorded distance. Theevaluation device 124 may be configured for removing influences frombackground light from the beam profile, for example, by an imagingwithout illumination.

The evaluation device 124 may be configured for determining thelongitudinal coordinate z_(DPR) for each of the reflection features byusing depth-from-photon-ratio technique. With respect todepth-from-photon-ratio (DPR) technique reference is made to WO2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the fullcontent of which is included by reference.

The evaluation device 124 may be configured for determining the beamprofile of each of the reflection features. The determining the beamprofile may comprise identifying at least one reflection featureprovided by the optical sensor 118 and/or selecting at least onereflection feature provided by the optical sensor 118 and evaluating atleast one intensity distribution of the reflection feature. As anexample, a region of the image may be used and evaluated for determiningthe intensity distribution, such as a three-dimensional intensitydistribution or a two-dimensional intensity distribution, such as alongan axis or line through the image. As an example, a center ofillumination by the light beam may be determined, such as by determiningthe at least one pixel having the highest illumination, and across-sectional axis may be chosen through the center of illumination.The intensity distribution may an intensity distribution as a functionof a coordinate along this cross-sectional axis through the center ofillumination. Other evaluation algorithms are feasible.

The analysis of the beam profile of one of the reflection features maycomprise determining at least one first area and at least one secondarea of the beam profile. The first area of the beam profile may be anarea A1 and the second area of the beam profile may be an area A2. Theevaluation device 124 may be configured for integrating the first areaand the second area. The evaluation device 123 may be configured toderive a combined signal, in particular a quotient Q, by one or more ofdividing the integrated first area and the integrated second area,dividing multiples of the integrated first area and the integratedsecond area, dividing linear combinations of the integrated first areaand the integrated second area. The evaluation device 124 may configuredfor determining at least two areas of the beam profile and/or to segmentthe beam profile in at least two segments comprising different areas ofthe beam profile, wherein overlapping of the areas may be possible aslong as the areas are not congruent. For example, the evaluation device124 may be configured for determining a plurality of areas such as two,three, four, five, or up to ten areas. The evaluation device 124 may beconfigured for segmenting the light spot into at least two areas of thebeam profile and/or to segment the beam profile in at least two segmentscomprising different areas of the beam profile. The evaluation device124 may be configured for determining for at least two of the areas anintegral of the beam profile over the respective area. The evaluationdevice 124 may be configured for comparing at least two of thedetermined integrals. Specifically, the evaluation device 124 may beconfigured for determining at least one first area and at least onesecond area of the beam profile. The first area of the beam profile andthe second area of the beam profile may be one or both of adjacent oroverlapping regions. The first area of the beam profile and the secondarea of the beam profile may be not congruent in area. For example, theevaluation device 124 may be configured for dividing a sensor region ofthe CMOS sensor into at least two sub-regions, wherein the evaluationdevice may be configured for dividing the sensor region of the CMOSsensor into at least one left part and at least one right part and/or atleast one upper part and at least one lower part and/or at least oneinner and at least one outer part.

Additionally or alternatively, the display device 110 may comprise atleast two optical sensors 118, wherein the light-sensitive areas of afirst optical sensor and of a second optical sensor may be arranged suchthat the first optical sensor is adapted to determine the first area ofthe beam profile of the reflection feature and that the second opticalsensor is adapted to determine the second area of the beam profile ofthe reflection feature. The evaluation device 124 may be adapted tointegrate the first area and the second area. T

In one embodiment, A1 may correspond to a full or complete area of afeature point on the optical sensor. A2 may be a central area of thefeature point on the optical sensor. The central area may be a constantvalue. The central area may be smaller compared to the full area of thefeature point. For example, in case of a circular feature point, thecentral area may have a radius from 0.1 to 0.9 of a full radius of thefeature point, preferably from 0.4 to 0.6 of the full radius.

The evaluation device 124 may be configured to derive the quotient Q byone or more of dividing the first area and the second area, dividingmultiples of the first area and the second area, dividing linearcombinations of the first area and the second area. The evaluationdevice 124 may be configured for deriving the quotient Q by

$Q = \frac{\int{\int_{A1}{{E\left( {x,y} \right)}{dxdy}}}}{\int{\int_{A2}{{E\left( {x,y} \right)}{dxdy}}}}$

wherein x and y are transversal coordinates, A1 and A2 are the first andsecond area of the beam profile, respectively, and E(x,y) denotes thebeam profile.

The evaluation device 124 may be configured for using at least onepredetermined relationship between the quotient Q and the longitudinalcoordinate for determining the longitudinal coordinate. Thepredetermined relationship may be one or more of an empiricrelationship, a semi-empiric relationship and an analytically derivedrelationship. The evaluation device may comprise at least one datastorage device for storing the predetermined relationship, such as alookup list or a lookup table.

The evaluation device 124 may be configured for executing at least onedepth-from-photon-ratio algorithm which computes distances for allreflection features with zero order and higher order.

The evaluation of the first image comprises sorting the identifiedreflection features with respect to brightness. The sorting may compriseassigning a sequence of the reflection features for further evaluationwith respect to brightness, in particular starting with the reflectionfeature having maximum brightness and subsequent the reflection featureswith decreasing brightness. The robustness of the determining of thelongitudinal coordinate z_(DPR) can be increased if the brightestreflection features are preferred for DPR computation. This is mainlybecause reflection features with zero order of diffraction grating arealways brighter than false features with a higher order.

The evaluation device 124 is configured for unambiguously matching ofreflection features with corresponding illumination features by usingthe longitudinal coordinate z_(DPR). The longitudinal coordinatedetermined with the depth-from-photon-ratio technique can be used forsolving the so called correspondence problem. In that way, distanceinformation per reflection feature can be used to find thecorrespondence of the known laser projector grid.

The illumination feature corresponding to the reflection feature may bedetermined using epipolar geometry. For description of epipolar geometryreference is made, for example, to chapter 2 in X. Jiang, H. Bunke:Dreidimensionales Computersehen” Springer, Berlin Heidelberg, 1997.Epipolar geometry may assume that an illumination image, i.e. an imageof the non-distorted illumination pattern, and the first image may beimages determined at different spatial positions and/or spatialorientations having a fixed distance. The distance may be a relativedistance, also denoted as baseline. The illumination image may be alsodenoted as reference image. The evaluation device 124 may be adapted todetermine an epipolar line in the reference image. The relative positionof the reference image and first image may be known. For example, therelative position of the reference image and the first image may bestored within at least one storage unit of the evaluation device. Theevaluation device 124 may be adapted to determine a straight lineextending from a selected reflection feature of the first image to areal world feature from which it originates. Thus, the straight line maycomprise possible object features corresponding to the selectedreflection feature. The straight line and the baseline span an epipolarplane. As the reference image is determined at a different relativeconstellation from the first image, the corresponding possible objectfeatures may be imaged on a straight line, called epipolar line, in thereference image. The epipolar line may be the intersection of theepipolar plane and the reference image. Thus, a feature of the referenceimage corresponding to the selected feature of the first image lies onthe epipolar line.

Depending on the distance to the object of the scene having reflectedthe illumination feature, the reflection feature corresponding to theillumination feature may be displaced within the first image 122. Thereference image may comprise at least one displacement region in whichthe illumination feature corresponding to the selected reflectionfeature would be imaged. The displacement region may comprise only oneillumination feature. The displacement region may also comprise morethan one illumination feature. The displacement region may comprise anepipolar line or a section of an epipolar line. The displacement regionmay comprise more than one epipolar line or more sections of more thanone epipolar line. The displacement region may extend along the epipolarline, orthogonal to an epipolar line, or both. The evaluation device 124may be adapted to determine the illumination feature along the epipolarline. The evaluation device 124 may be adapted to determine thelongitudinal coordinate z for the reflection feature and an errorinterval ±£ from the combined signal Q to determine a displacementregion along an epipolar line corresponding to Z±E or orthogonal to anepipolar line. The measurement uncertainty of the distance measurementusing the combined signal Q may result in a displacement region in thesecond image which is non-circular since the measurement uncertainty maybe different for different directions. Specifically, the measurementuncertainty along the epipolar line or epipolar lines may be greaterthan the measurement uncertainty in an orthogonal direction with respectto the epipolar line or lines. The displacement region may comprise anextend in an orthogonal direction with respect to the epipolar line orepipolar lines. The evaluation device 124 may be adapted to match theselected reflection feature with at least one illumination featurewithin the displacement region. The evaluation device 124 may be adaptedto match the selected feature of the first image with the illuminationfeature within the displacement region by using at least one evaluationalgorithm considering the determined longitudinal coordinate z_(DPR).The evaluation algorithm may be a linear scaling algorithm. Theevaluation device 124 may be adapted to determine the epipolar lineclosest to and/or within the displacement region. The evaluation devicemay be adapted to determine the epipolar line closest to the imageposition of the reflection feature. The extent of the displacementregion along the epipolar line may be larger than the extent of thedisplacement region orthogonal to the epipolar line. The evaluationdevice 124 may be adapted to determine an epipolar line beforedetermining a corresponding illumination feature. The evaluation device124 may determine a displacement region around the image position ofeach reflection feature. The evaluation device 124 may be adapted toassign an epipolar line to each displacement region of each imageposition of the reflection features, such as by assigning the epipolarline closest to a displacement region and/or within a displacementregion and/or closest to a displacement region along a directionorthogonal to the epipolar line. The evaluation device 124 may beadapted to determine the illumination feature corresponding to thereflection feature by determining the illumination feature closest tothe assigned displacement region and/or within the assigned displacementregion and/or closest to the assigned displacement region along theassigned epipolar line and/or within the assigned displacement regionalong the assigned epipolar line.

Additionally or alternatively, the evaluation device 124 may beconfigured to perform the following steps:

-   -   Determining a displacement region for the image position of each        reflection feature;    -   Assigning an epipolar line to the displacement region of each        reflection feature such as by assigning the epipolar line        closest to a displacement region and/or within a displacement        region and/or closest to a displacement region along a direction        orthogonal to the epipolar line;    -   Assigning and/or determining at least one illumination feature        to each reflection feature such as by assigning the illumination        feature closest to the assigned displacement region and/or        within the assigned displacement region and/or closest to the        assigned displacement region along the assigned epipolar line        and/or within the assigned displacement region along the        assigned epipolar line.

Additionally or alternatively, the evaluation device 124 may be adaptedto decide between more than one epipolar line and/or illuminationfeature to be assigned to a reflection feature such as by comparingdistances of reflection features and/or epipolar lines within theillumination image and/or by comparing error weighted distances, such as6-weighted distances of illumination features and/or epipolar lineswithin the illumination image and assigning the epipolar line and/orillumination feature in shorter distance and/or 6-weighted distance tothe illumination feature and/or reflection feature.

As outlined above, due to diffraction grating a plurality of reflectionfeatures, e.g. for each illumination feature one real feature and aplurality of false features, are generated. The matching is performedwith decreasing brightness of the reflection features starting with thebrightest reflection feature. No other reflection feature can beassigned to the same matched illumination feature. In due of the displayartifacts, the false features which are generated are generally darkerthan the real features. By sorting the reflection features bybrightness, brighter reflection features are preferred for thecorrespondence matching. If a correspondence of an illumination featureis already used, a false feature cannot be assigned to a used, i.e.matched, illumination feature.

FIG. 2A shows a simulated first image 122 without the display 112 for anillumination pattern comprising a single light spot. FIG. 2B shows afirst image 122 captured by the optical sensor 118 behind display 112.It is observed that the diffraction grating generate multiple spots. InFIG. 2B the real feature is shown as reference number 126 and exemplarya false feature is shown as reference number 128. FIG. 2C shows afurther example of a first image 122 captured by the optical sensor 118behind display 112, wherein in this case the illumination pattern is aprojected laser grid. Multiple spots appear in due of diffractiongrating.

FIG. 3A shows a further exemplary first image 122 of a scene withprojected laser spots. Reflection feature of zero order of diffractiongrating 130 and of higher order 132 are shown. FIGS. 3B and 3C showmatching of reflection features and illumination features. On the leftpart of FIGS. 3B and 3C the first image 122 is shown and on the rightpart a corresponding illumination pattern, comprising two illuminationfeatures. The first image 122 may comprise six reflection features. Theevaluation device 124 may be configured for identifying the reflectionfeatures in the first image 122 and to sort them with respect to theirbrightness. As shown in FIG. 3B two of the reflection features may bebrighter compared to the other reflection features. The evaluationdevice 124 may start beam profile analysis and matching with anillumination feature with one of the two brighter reflection features,denoted with circle 134. Each of two brighter reflection features may bematched with one illumination feature, denoted with an arrow. Theevaluation device 124 may classify said matched features as realfeatures. As depicted in FIG. 3C, the two illumination features of theillumination pattern are already matched with the brighter reflectionfeatures. No other reflection feature can be assigned to the samematched illumination feature. By sorting the reflection features bybrightness, brighter reflection features are preferred for thecorrespondence matching. If a correspondence of an illumination featureis already used, a false feature cannot be assigned to a used, i.e.matched, illumination feature. Thus, the two remaining reflectionfeatures, denoted with circles 136, have no corresponding illuminationfeature and cannot be assigned to any point of the pattern. Saidremaining reflection features are classified by the evaluation device124 as false features.

The evaluation device 124 is configured for rejecting the false featuresand for generating a depth map for the real features by using thelongitudinal coordinate z_(DPR). The display device 110 may be used togenerate a 3D map from a scene, e.g. of a face.

The depth map can be further refined by using a further depthmeasurement technique such as triangulation and/or depth-from-defocusand/or structured light. The evaluation device may be-configured fordetermining at least one second longitudinal coordinate z_(triang) foreach of the reflection features using triangulation and/ordepth-from-defocus and/or structured light techniques. The evaluationdevice 124 may be configured for determining a combined longitudinalcoordinate of the second longitudinal coordinate z_(triang) and thelongitudinal coordinate z_(DPR). The combined longitudinal coordinatemay be a mean value of the second longitudinal coordinate z_(triang) andthe longitudinal coordinate z_(DPR). The combined longitudinalcoordinate may be used for determining the depth map.

As shown in FIG. 1B, the display device 110 may comprise a furtherillumination source 138. The further illumination source 138 maycomprise at least one light emitting diode (LED). The furtherillumination source 138 may be configured for generating light in thevisual spectral range. The optical sensor 118 may be configured fordetermining at least one second image comprising at least one twodimensional image of the scene. The further illumination source 138 maybe configured for providing additional illumination for imaging of thesecond image. For example, the setup of the display device 110 can beextended by an additional flood illumination LED. The furtherillumination source 138 may illuminate the scene, such as a face, withthe LED and, in particular, without the illumination pattern, and theoptical sensor 118 may be configured for capturing the two-dimensionalimage. The 2D image may be used for face detection and verificationalgorithm.

The distorted image captured by the optical sensor 118 can be repaired,if an impulse response of the display 112 is known. The evaluationdevice 124 may be configured for determining at least one correctedimage I₀ by deconvoluting the second image I with a grating function g,wherein I=1₀*g. The grating function is also denoted impulse response.The undistorted image can be restored by a deconvolution approach, e.g.Van-Cittert or Wiener Deconvolution.

As shown in FIG. 4 , the display device 110 may be configured fordetermining the grating function g. The display device 110 may beconfigured for illuminating a black scene with an illumination patterncomprising a small single bright spot, denoted with reference number140. The captured image 142 may be the grating function. This proceduremay be performed only once such as during calibration. For determining acorrected image even for imaging through the display 112, the displaydevice 110 may be configured for capturing the image and use thedeconvolution approach with the captured impulse response g. Theresulting image may be a reconstructed image with less artifacts of thedisplay and can be used for several applications, e.g. face recognition.FIGS. 5A to 5C show examples of two-dimensional images captured with theoptical sensor 118. In FIG. 5A the exemplary scene was captured with theoptical sensor 118 behind the display 112. In FIG. 5B the exemplaryscene was captured with the optical sensor 118 without the display 112.FIG. 5C shows the reconstructed image with the deconvolution approach.

LIST OF REFERENCE NUMBERS

-   110 display device-   112 display-   114 illumination source-   116 housing-   118 optical sensor-   120 light-sensitive area-   122 first image-   124 evaluation device-   126 real feature-   128 false feature-   130 zero order of diffraction grating-   132 higher order-   134 circle-   136 circle-   138 further illumination source-   140 for illuminating a black scene-   142 captured image

CITED REFERENCES

-   DE 20 2018 003 644 U1-   U.S. Pat. No. 9,870,024 B2-   U.S. Pat. No. 10,057,541 B2-   U.S. Pat. No. 10,215,988 B2-   WO 2018/091649 A1-   WO 2018/091638 A1-   WO 2018/091640 A1-   WO 2019/042956 A1

1. A display device comprising at least one illumination sourceconfigured for projecting at least one illumination pattern comprising aplurality of illumination features on at least one scene; at least oneoptical sensor having at least one light sensitive area, wherein theoptical sensor is configured for determining at least one first imagecomprising a plurality of reflection features generated by the scene inresponse to illumination by the illumination features; and at least onetranslucent display configured for displaying information, wherein theillumination source and the optical sensor are placed in a direction ofpropagation of the illumination pattern in front of the display; atleast one evaluation device, wherein the evaluation device is configuredfor evaluating the first image, wherein evaluating the first imagecomprises identifying the reflection features of the first image andsorting the identified reflection features with respect to brightness,wherein each of the reflection features comprises at least one beamprofile, wherein the evaluation device is configured for determining atleast one longitudinal coordinate z_(DPR) for each of the reflectionfeatures by analysis of their beam profiles, wherein the evaluationdevice is configured for unambiguously matching reflection features withcorresponding illumination features by using the longitudinal coordinatez_(DPR), wherein the matching is performed with decreasing brightness ofthe reflection features starting with the brightest reflection feature,wherein the evaluation device is configured for classifying a reflectionfeature being matched with an illumination feature as a real feature andfor classifying a reflection feature not being matched with anillumination feature as a false feature, wherein the evaluation deviceis configured for rejecting the false features and for generating adepth map for the real features by using the longitudinal coordinatez_(DPR).
 2. The display device according to claim 1, wherein theevaluation device is configured for determining at least one secondlongitudinal coordinate z_(triang) for each of the reflection featuresusing triangulation and/or depth-from-defocus and/or structured lighttechniques.
 3. The display device according to claim 2, wherein theevaluation device is configured for determining a combined longitudinalcoordinate of the second longitudinal coordinate z_(triang) and thelongitudinal coordinate z_(DPR), wherein the combined longitudinalcoordinate is a mean value of the second longitudinal coordinatez_(triang) and the longitudinal coordinate z_(DPR), wherein the combinedlongitudinal coordinate is used for generating the depth map.
 4. Thedisplay device according to claim 1, wherein the illumination sourcecomprises at least one laser projector, wherein the laser projectorcomprises at least one laser source and at least one diffractive opticalelement (DOE).
 5. The display device according to claim 1, wherein theillumination source is configured for generating at least one light beamhaving a beam path passing from the illumination source through thedisplay to the scene, wherein the display is configured for functioningas grating such that the light beam experiences diffraction by thedisplay which results in the illumination pattern.
 6. The display deviceaccording to claim 5, wherein a wiring of the display is configured forforming gaps and/or slits and ridges of the grating.
 7. The displaydevice according to claim 1, wherein the illumination pattern comprisesa periodic point pattern.
 8. The display device according to claim 1,wherein the illumination pattern has a low point density, wherein theillumination pattern has ≤2500 points per field of view.
 9. The displaydevice according to claim 1, wherein the evaluation device is configuredfor determining the beam profile information for each of the reflectionfeatures by using depth-from-photon-ratio technique.
 10. The displaydevice according to claim 1, wherein the optical sensor comprises atleast one CMOS sensor.
 11. The display device according to claim 1,wherein the display device comprises a further illumination source,wherein the further illumination source comprises at least one lightemitting diode (LED).
 12. The display device according to claim 11,wherein the further illumination source is configured for generatinglight in the visual spectral range.
 13. The display device according toclaim 11, wherein the optical sensor is configured for determining atleast one second image comprising at least one two dimensional image ofthe scene, wherein the further illumination source is configured forproviding additional illumination for imaging of the second image. 14.The display device according to claim 13, wherein the evaluation deviceis configured for determining at least one corrected image I₀ bydeconvoluting the second image I with a grating function g, whereinI=I₀*g.
 15. A method for depth measurement through a translucentdisplay, wherein at least one display device according to claim 1 isused, wherein the method comprises the following steps: a) projecting atleast one illumination pattern comprising a plurality of illuminationfeatures on at least one scene by using at least one illuminationsource, wherein the illumination source is placed in the direction ofpropagation of the illumination pattern in front of the display; b)determining at least one first image comprising a plurality ofreflection features generated by the scene in response to illuminationby the illumination features by using at least one optical sensor,wherein the optical sensor has at least one light sensitive area,wherein the optical sensor is placed in the direction of propagation ofthe illumination pattern in front of the display, wherein each of thereflection features comprises at least one beam profile; c) Evaluatingthe first image by using at least one evaluation device, whereinevaluating the first image comprises the following substeps: C1)identifying the reflection features of the first image and sorting theidentified reflection features with respect to brightness; C2)determining at least one longitudinal coordinate z_(DPR) for each of thereflection features by analysis of their beam profiles; C3)unambiguously matching reflection features with correspondingillumination features by using the longitudinal coordinate z_(DPR),wherein the matching is performed with decreasing brightness of thereflection features starting with the brightest reflection feature; C4)classifying a reflection feature being matched with an illuminationfeature as a real feature and classifying a reflection feature not beingmatched with an illumination feature as a false feature; and C5)rejecting the false features and generating the depth map for the realfeatures by using the longitudinal coordinate z_(DPR).
 16. A method ofusing the display device according to claim 1 for a purpose selectedfrom the group consisting of: a position measurement in traffictechnology; an entertainment application; a security application; asurveillance application; a safety application; a human-machineinterface application; a tracking application; a photographyapplication; an imaging application or camera application; a mappingapplication for generating maps of at least one space; a homing ortracking beacon detector for vehicles; an outdoor application; a mobileapplication; a communication application; a machine vision application;a robotics application; a quality control application; and amanufacturing application.